http requests

how computers ask nicely

quick recap

In middleware we learned that middleware functions:
- Receive a request
- Send a response
Middleware is what happens between those two things.

Setting (low-level) expectations

Node's HTTP built-in module
Other, more useful modules (such as Axios)
Connecting to third party APIs & databases
- The Service Pattern
Connecting to anything
- The Repository Pattern

Before we begin

Fork & Clone
https://github.com/brianboyko/httprequests-patterns
if you haven't already

$ git checkout master

$ git checkout -b develop

$ yarn

$ yarn setupenv

Everything's done

Unlike the previous course, we're not going to be going through livecoding examples until the end.

The codebase is mostly to

point out how different patterns are used - to actually build these patterns would take hours

Express

We'll be using Express.js to handle our servers. For the most part we won't be doing that much, though we will be writing new middleware to handle requests.

NeDB

This repo uses NeDB, which, basically, is a lightweight version of MongoDB, and it's easy to set up. (No need to install extra software or run a daemon). Keep in mind, different databases will require different approaches.

import http from 'http';

and

import https from 'https';

Node has a built-in http (and https) request module.
We probably won't ever use it, but all the other request libraries are based upon it.
Let's do an experiment

import https from 'https';

Create /src/externalApi/builtInVsLibraries/builtInHttps.js

// ./src/externalApi/builtInVsLibraries/builtInHttps.js
import https from "https"; // we don't need to yarn add https becase https is built into node!
// if we were using http, import "http"; 

export const getFirstPost = () =>
  new Promise((resolve, reject) => {
    // we define the http options; 
    const httpsOptions = {
      method: "GET",
      hostname: "jsonplaceholder.typicode.com",
      path: "/posts/1",
      headers: {
        Accept: "application/json",
      },
    };
    const req = https.request(httpsOptions, (res) => {
      // req and res are not objects, they are 'streams'; 
      const dataArray = [];
      res.setEncoding("utf8"); // without this, we'll get a Buffer, not string.
      res.on("data", (chunk) => {
        // data can often get sent in multiple chunks. 
        dataArray.push(chunk);
      });
      res.on("end", () => {
        // join the data strings, parse them from JSON into object, then resolve.
        resolve(JSON.parse(dataArray.join("")));
      });
    });
    req.on("error", (err) => {
      console.error(err);
      reject(err);
    });
    req.end(); // if we don't end, the request will never figure out we're done adding stuff.
  });

import axios from 'axios';

Axios is just *one* library out of many, including request, fetch, isomorphic-fetch, isomorphic-unfetch, superagent (front end only), etc.

// ./src/externalApi/builtInVsLibraries/axiosBased.js

import axios from "axios";

export const getFirstPostViaAxios = () =>
  new Promise((resolve, reject) =>
    axios
      .get(`https://jsonplaceholder.typicode.com/posts/1`)
      .then((result) => {
        resolve(result.data);
      })
      .catch(reject)
  );

99% of the time, you'll be using a request library like axios, for obvious reasons.

let's test this puppy

We'll be using jest for this.

create ./src/externalApi/builtInVsLibraries/index.spec.js

// ./src/externalApi/builtInVsLibraries/index.spec.js

import { getFirstPost } from "./builtInHttps";
import { getFirstPostViaAxios } from "./axiosBased";

const EXPECTED_RESULT = {
  userId: 1,
  id: 1,
  title:
    "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  body:
    "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto",
};

describe("Getting the first post", () => {
  describe("https module", () => {
    it("gets the expected result using node's built-in https", async () => {
      const result = await getFirstPost();
      expect(result).toEqual(EXPECTED_RESULT);
    });
  });
  describe("with Axios", () => {
    it("gets the expected result using axios", async () => {
      const result = await getFirstPostViaAxios();
      expect(result).toEqual(EXPECTED_RESULT);
    });
  });
});

post is pretty much the same.

// ./src/externalApi/builtInVsLibraries/axiosBased.js

import axios from "axios";

export const getFirstPostViaAxios = () =>
  new Promise((resolve, reject) =>
    axios
      .get(`https://jsonplaceholder.typicode.com/posts/1`)
      .then((result) => {
        resolve(result.data);
      })
      .catch(reject)
  );

/* NEW STUFF */
export const postToApi = (data) =>
  new Promise((resolve, reject) =>
    axios
      .post(`https://jsonplaceholder.typicode.com/posts`, data, {
        headers: {
          "Content-type": "application/json; charset=UTF-8",
        },
      })
      .then((result) => resolve(result.data))
      .catch(reject)
  );
/* END NEW STUFF */

let's test this puppy

./src/externalApi/builtInVsLibraries/index.spec.js

// ./src/externalApi/builtInVsLibraries/index.spec.js

import { getFirstPost } from "./builtInHttps";
import { getFirstPostViaAxios , /*new!*/ postToApi } from "./axiosBased";

const EXPECTED_RESULT = {
  userId: 1,
  id: 1,
  title:
    "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  body:
    "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto",
};

describe("Getting the first post", () => {
 /* ... */
});

// NEW!
describe("Using Axios to post", () => {
  it("makes a post request", async () => {
    const result = await postToApi({ title: "foo", body: "bar", userId: 1 });
    expect(result).toEqual({ title: "foo", body: "bar", userId: 1, id: 101 });
  });
});

i believe in you

if you want to find out more about other methods, such as patch, put, etc. or other libraries, you're just a quick google away.

But let's talk about best practices.

This is Jeopardy!

Let's access a third-party API service

jservice.io

create /src/externalApi/jService directory

The Service Pattern

organizing your external api

const ChatApplication = {
  login: (username, password) => {},
  logout: (user) => {},
  joinChatRoom: (roomId, user) => {},
  sendMessage: (message, roomId, user) => {},
  sendPrivateMessage: (message, recieverId, user) => {}
}

export default ChatApplication;

Don't overthink it.

The service pattern just provides a unified object with methods designed to interact with the API automatically.

What do we want to do?

We want to get a list of all the categories with IDs*
- * and save it to a database, but this comes later.
We want to get a list of all the questions for a given category
Maybe we want to combine the two - get a number of categories AND the questions for all those categories

let's get some categories

./src/externalApi/jService/categories.js

// ./src/externalApi/jService/categories.js

import axios from "axios";

const JSERVICE_URL = `http://jservice.io`;

export const getCategories = async (count = 10, offset = 0) => {
  try {
    // we could do this as 
    // axios.get(`${JSERVICE_URL}/api/categories/?count=${count}&offset=${offset}`) 
    // too!
    const result = await axios.get(`${JSERVICE_URL}/api/categories/`, {
      params: { count, offset },
    });
    return result.data;
  } catch (err) {
    throw new Error(err);
  }
};

let's test this small dog

// ./src/externalApi/jService/categories.spec.js
import { getCategories } from "./categories";

describe("jService categories", () => {
  it("gets the categories", async () => {
    const result = await getCategories(10, 0);
    expect(result).toHaveLength(10);
    expect(result.map((cat) => `${cat.id}:${cat.title}`)).toEqual([
      "11531:mixed bag",
      '11532:let\'s "ch"at',
      "5412:prehistoric times",
      "11496:acting families",
      "11498:world city walk",
      "11499:tough-pourri",
      "11500:visualliteration",
      "11504:quotations from bartlett's",
      "11542:fill in the history _____",
      "11544:the state it's in",
    ]);
  });
  it("gets the next five categories", async () => {
    const result = await getCategories(5, 10);
    expect(result).toHaveLength(5);
    expect(result.map((cat) => `${cat.id}:${cat.title}`)).toEqual([
      '11521:"hot" stufff',
      "7580:animal words & phrases",
      "11522:you're in this foreign country if...",
      "11523:the secret lives of teachers",
      "11512:harry truman",
    ]);
  });
});

./src/externalApi/jService/category.js

// ./src/externalApi/jService/categories.js

import axios from "axios";

const JSERVICE_URL = `http://jservice.io`;

export const getCategories = async (count = 10, offset = 0) => {
  try {
    // we could do this as 
    // axios.get(`${JSERVICE_URL}/api/categories/?count=${count}&offset=${offset}`) 
    // too!
    const result = await axios.get(`${JSERVICE_URL}/api/categories/`, {
      params: { count, offset },
    });
    return result.data;
  } catch (err) {
    throw new Error(err);
  }
};

Harry Truman: Category #11512

import { getCategoryById } from "./category";

const HARRY_TRUMAN_CAT_ID = 11512;

describe("jService category (getCategoryById)", () => {
  it("gets the categoryById", async () => {
    const result = await getCategoryById(HARRY_TRUMAN_CAT_ID);
    expect(result.id).toBe(11512);
    expect(result.title).toBe("harry truman");
    expect(result.clues_count).toBe(5);
    expect(result.clues).toHaveLength(5);
  });
});

Organizing this into a Service Pattern

/src/externalApi/jService/index.js

import { getCategories } from "./categories";
import { getCategoryById } from "./category";

// let's get categories AND request the clues for those categories.
const getCategoriesAndClues = async (count, offset) => {
  try {
    const categories = await getCategories(count, offset);
    const categoriesWithClues = await Promise.all(
      categories.map(async (category) => {
        const cluesResult = await getCategoryById(category.id);
        return { ...category, clues: cluesResult.clues };
      })
    );
    return categoriesWithClues;
  } catch (err) {
    throw new err();
  }
}

const jService = {
  getCategories,
  getCategoryById,
  getCategoriesAndClues
};

export default jService;

let's quiz this canine

import jService from "./index";

describe("jService", () => {
  it("has three methods we can use", () => {
    expect(jService).toHaveProperty("getCategories");
    expect(jService).toHaveProperty("getCategoryById");
    expect(jService).toHaveProperty("getCategoriesAndClues");
  });
  it("will get categories and clues via jService.getCategoriesAndClues", async () => {
    const threeCategories = await jService.getCategoriesAndClues(3, 14);
    expect(threeCategories).toHaveLength(3);
    expect(threeCategories[0].title).toBe("harry truman");
    expect(threeCategories[0].clues).toHaveLength(5);
    expect(threeCategories[0].clues[0].question).toBe(
      "In 1945 Harry served just 83 days in this job"
    );
    expect(threeCategories[1].title).toBe("contract killings");
    expect(threeCategories[1].clues).toHaveLength(5);
    expect(threeCategories[1].clues[0].question).toBe(
      "If the show's named after you, they can't kill you off over a contract spat, can they?  Hmm... ask this \"Valerie\" star"
    );
    expect(threeCategories[2].title).toBe("sham, wow!");
    expect(threeCategories[2].clues).toHaveLength(5);
    expect(threeCategories[2].clues[0].question).toBe(
      "In 1925 Victor Lustig sold this 984-foot Paris landmark--twice!  Now that's a bargain for Victor at twice the price!"
    );
  });
});

database service

prep:

create a blank ./data/jeopardy.db file
create a blank ./data/test.db file
edit ./src/db/index.js

// ./src/db/index.js;
import Datastore from "nedb";
import path from "path";

const data = new Datastore({
  filename: path.resolve(__dirname, "../../data/data.db"),
  autoload: true,
});

const jeopardy = new Datastore({
  filename: path.resolve(__dirname, "../../data/jeopardy.db"),
  autoload: true,
});
jeopardy.ensureIndex({ fieldName: "id", unique: true }, function (err) {
  if (err) {
    throw new Error(err);
  }
});

const test = new Datastore({
  filename: path.resolve(__dirname, "../../data/test.db"),
  autoload: true,
});
test.ensureIndex({ fieldName: "id", unique: true }, function (err) {
  if (err) {
    throw new Error(err);
  }
});

export default { data, jeopardy, test };

saving jeopardy

A more complex service for the DB.

./src/db/jeopardy/index.js

// ./src/db/jeopardy/index.js;
import db from "../index"; // let's get the NeDB database.
import addCategory from "./addCategory";
import findCategory from "./findCategory";
import searchCategories from "./searchCategories";
import updateCategory from "./updateCategory";
import deleteCategory from "./deleteCategory";

export const dbService = (collection) => {
  return {
    create: {
      category: addCategory(collection),
      categories: addCategory(collection), // quirk of syntax.
    },
    read: {
      category: findCategory(collection),
      categoryByTitle: (title) => findCategory(collection)({ title }),
      categoryById: (id) => findCategory(collection)({ id }),
      searchCategoriesByRegex: searchCategories(collection),
      searchCategories: (substring) =>
        searchCategories(collection)(new RegExp(substring)),
    },
    update: {
      category: updateCategory(collection),
    },
    delete: {
      category: deleteCategory(collection),
      categoryById: (id) => deleteCategory(collection)({ id }),
      categoryByTitle: (title) => deleteCategory(collection)({ title }),
    },
  };
};

export default dbService(db.jeopardy);

why curry?

allows us to provide different databases for testing, development, and production

allows us to test components individually using stubs and mocks.

import dbService from './index'
returns the service connected to the Jeopardy collection

import {dbService} from './index'

returns a function that takes a collection and returns a service connected to the provided collection

let's challenge this woofer

import { dbService } from "./index";
import db from "../index";
import mockCategories from "./__mockdata__/categories.json";
import mockCategoriesAndClues from "./__mockdata__/categories_clues.json";

const testDb = dbService(db.test);

describe("dbService", () => {
  beforeAll(() => {
    testDb.drop();
  });
  describe("create", () => {
    describe("dbService.create.category", () => {
      it("creates a category", async () => {
        const result = await testDb.create.category(mockCategories[0]);
        expect(result).toEqual({
          _id: result._id,
          clues_count: 5,
          id: 11512,
          title: "harry truman",
        });
      });
      // AND SO ON AND SO ON

combining data sources

the repository pattern

The repository exposes public methods. These methods are asynchronous (uses callbacks or returns Promises).
Whether the source is:
- static content,
- external services
- databases
is not important to the calling function. The calling component should not need to know anything about it.

otherwise, it's basically like our services pattern

localhost:3000/jeopardy/:categoryId

using the jService (external API)
using the dbService (local database)
using a repository pattern
using middleware

./src/repos/jeopardy.js

import jService from "../externalApi/jService";
import db from "../db/index"; // let's get the NeDB database.
import { dbService } from "../db/jeopardy/index";

const getCategoryById = (databaseService) => async (id) => {
  const dbResults = await databaseService.read.categoryById(id);
  if (dbResults.length === 0) {
    console.info(`Cache miss for id# ${id}`);
    const categoryData = await jService.getCategoryById(id);
    databaseService.create.category(categoryData); // will run in background.
    return { ...categoryData, cache: "miss" };
  }
  console.info(`Cache hit for id# ${id}`);
  return dbResults[0];
};

const getCategoryByTitle = (databaseService) => async (title) => {
  const dbResults = await databaseService.read.categoryByTitle(title);
  if (dbResults.length === 0) {
    throw new Error(`No results for title: ${title} in db`);
  }
  return dbResults[0];
};

export const jeopardy = (collection) => {
  return {
    getCategory: (idOrTitle) => {
      const parsed = parseInt(idOrTitle, 10);
      return isNaN(parsed)
        ? getCategoryByTitle(dbService(collection))(idOrTitle)
        : getCategoryById(dbService(collection))(parsed);
    },
    getCategoryBypassCache: async (id) => {
      const result = await jService.getCategoryById(id);
      return { ...result, cache: "bypassed" };
    },
  };
};

export default jeopardy(db.jeopardy);

lets -- you know, this joke has gotten old

import { jeopardy } from "./jeopardy";
import db from "../db/index";
import { dbService } from "../db/jeopardy/index";

const testDb = dbService(db.test);
const testRepo = jeopardy(db.test);

describe("jeopardy repository", () => {
  beforeAll(() => {
    testDb.drop();
  });
  describe("getCategoryBypassCache", () => {
    it("gets a repo directly from the api", async () => {
      const result = await testRepo.getCategoryBypassCache(11512);
      expect(result.cache).toBe("bypassed");
      expect(result.clues).toHaveLength(5);
      expect(result.title).toBe("harry truman");
    });
  });
  describe("getCategory", () => {
    describe("gets a category, but does'nt really care where from", () => {
      it("will grab from the API on a cache miss", async () => {
        const result = await testRepo.getCategory("11512");
        expect(result.cache).toBe("miss");
        expect(result.clues).toHaveLength(5);
        expect(result.title).toBe("harry truman");
      });
      it("will grab from the API on a cache hit", async () => {
        const result = await testRepo.getCategory("11512");
        expect(result.cache).toBe(undefined);
        expect(result.clues).toHaveLength(5);
        expect(result.title).toBe("harry truman");
      });
      it("will grab from the database by title if in the DB", async () => {
        const result = await testRepo.getCategory("harry truman");
        expect(result.cache).toBe(undefined);
        expect(result.clues).toHaveLength(5);
        expect(result.title).toBe("harry truman");
      });
      it("will error if searching by title and not in the DB", async () => {
        try {
          const result = await testRepo.getCategory("suck it, trebek!");
          expect(true).toBeFalse(); // this line should not run.
        } catch (err) {
          expect(err.toString()).toBe(
            "Error: No results for title: suck it, trebek! in db"
          );
        }
      });
    });
  });
});

putting it all together

import { jeopardy } from "../../repos/jeopardy";
import db from "../../db/index";

const jeopardyMiddleware = async (req, res, next) => {
  const { identifier } = req.params;
  const { test } = req.query;
  const repo = test ? jeopardy(db.test) : jeopardy(db.jeopardy);
  try {
    const result = await repo.getCategory(identifier);
    res.send(result);
  } catch (err) {
    next(err);
  }
};

export default jeopardyMiddleware;

./src/server/middleware/jeopardy.js

  app.get("/jeopardy/:identifier", jeopardyMiddleware);

./src/server/index.js - add one line

import axios from "axios"; // we'll learn more about Axios later.
import launchServer from "./index";
import db from "../db/index";
import mockData from "../__mockdata__/categories_clues.json";

const TEST_URL = "http://localhost";
const TEST_PORT = 3434;
let server;

describe("/src/server/index.js", () => {
  beforeAll(async () => {
    // to test the server, we have to launch the server
    server = launchServer(TEST_PORT);
    // let's also drop the test db
    await db.test.remove({}, { multi: true }, function (err, numRemoved) {
      if (err) {
        throw new Error(err);
      }
      console.log(`Successfully removed ${numRemoved} documents`);
    });
    return;
  });
  afterAll(() => {
    // we returned the server object from launchServer so that we can close it;
    server.close();
    return;
  });
  // Let's test our endpoints
  describe("get /hello-world", () => {
    it("gets hello world", async () => {
      const result = await axios.get(`${TEST_URL}:${TEST_PORT}/hello-world`);
      expect(result.data).toBe("Hello World!");
    });
  });
  describe("post /hello-world", () => {
    it("posts hello world", async () => {
      const result = await axios.post(`${TEST_URL}:${TEST_PORT}/hello-world`);
      expect(result.data).toBe("Hello World From Post!");
    });
  });
  describe("get /jeopardy/:identifier", () => {
    it("gets a category from jeopardy", async () => {
      // This WILL access db.jeopardy, not db.test
      const firstDip = await axios.get(
        `${TEST_URL}:${TEST_PORT}/jeopardy/11512`,
        { params: { test: true } }
      );
      expect(firstDip.data).toEqual({
        ...mockData[0],
        cache: "miss",
        _id: firstDip.data._id,
      });
    });
    it("grabs from the cache on a second dip.", async () => {
      // This WILL access db.jeopardy, not db.test
      const secondDip = await axios.get(
        `${TEST_URL}:${TEST_PORT}/jeopardy/11512`,
        { params: { test: true } }
      );
      expect(secondDip.data).toEqual({
        ...mockData[0],
        cache: undefined,
        _id: secondDip.data._id,
      });
    });
  });
});

./src/server/index.spec.js

questions & experiments

HTTP requests

By brianboyko

http requests

how computers ask nicely

quick recap

Setting (low-level) expectations

Before we begin

Everything's done

Express

NeDB

import http from 'http';

import https from 'https';

import https from 'https';

import axios from 'axios';

let's test this puppy

post is pretty much the same.

let's test this puppy

i believe in you

This is Jeopardy!

jservice.io

The Service Pattern

organizing your external api

What do we want to do?

let's get some categories

./src/externalApi/jService/categories.js

let's test this small dog

./src/externalApi/jService/category.js

Harry Truman: Category #11512

Organizing this into a Service Pattern

let's quiz this canine

database service

saving jeopardy

why curry?

let's challenge this woofer

combining data sources

the repository pattern

localhost:3000/jeopardy/:categoryId

putting it all together

HTTP requests

More from brianboyko