Robert Roskam
Engineer Manager at Pantheon
How to Sanely Test Complex Systems
test coverage
100% is too little
//inverse.js
function inverse(x) {
return 1 / x;
}
// inverse.test.js
describe("inverse", () => {
it("should return 1 / x", () => {
expect(inverse(4)).toBe(0.25);
expect(inverse(-8)).toBe(-0.125);
});
});100% Test Coverage
Uber's service topology
src: https://www.uber.com/blog/microservice-architecture/
Possible Solutions
Concept: Benefit
Test Seeds: stable data
Test Clocks: manageable time
Test Setups: repeatable state setting processes
My Assumptions About You
- You have made manual tests in a system
- You have experienced the time drain that is doing this by hand every week
- You've attempted end-to-end integration tests
- You've wondered if there's a better way
Test Seed
Test seeds are unchanging data objects in your system.
Any interactions with them give the same answer every time without fail.
Test Seed
GET /users/42
{ user_id: 42, balance: 10, ... }
GET /users/66
{ user_id: 66, balance: 0, ... }POST /charge/
{amount: 1, used_id: 42}
Response 201
{id: 111}
POST /charge/
{amount: 1, used_id: 66}
Response 400
{"error": "insufficient funds"}
- Stripe test cards
- e.g. 4242424242424242
- Twilio magic numbers
- e.g. +1-500-555-0006
Test Seed
Test Seed
Examples in the Wild
Make time an object in your system.
Stop it for an object, a user, or the entire system.
Move it forward by hours or days.
Test Clock
Test Clock
POST /test-clock/
{ name: "1st of the Month", frozenTime: "2023-08-01" }
Response 201
{id: 111}
POST /gift-card/
{ id: 77, ..., expiresAt: "2024-07-31", testClockID: 111}
POST /test-clock/111/advance/
{ advanceTo: "2024-08-01"}
GET /gift-card/77
{ id: 77, ..., status: 'expired'}Test Clock
Examples in the Wild
Test Setups
Make APIs to run various steps to put your system into a desired state.
Test Setups
POST /user/mock-bulk/
{ count: 10 }
Response 201
[
{id: 110, name: "Joe Small", ...},
{id: 111, name: "Abby Zales", ...},
{id: 112, name: "Troy Barnes", ...},
...
]
Examples in the Wild
- Shopify Bogus Gateway
Robert Roskam
@raiderrobert
mastodon | github
Work at
How to Sanely Test Complex Systems
By Robert Roskam
