JSON Schemas are Your True Testing Friend

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

Gleb Bahmutov, PhD

VP of Engineering Cypress.io

Don't trust this guy

👉

Every time code crosses a border ...

Defensive coding: checking inputs before each computation (preconditions). Sometimes checking result value before returning (postconditions).

Paranoid coding: checking inputs before each computation, as if the caller was evil and trying to break the function on purpose.

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

predicate

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

explanation

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

explanation

🔑

STOP throwing Errors

START throwing Explanations

undefined is not a function

vs

What went wrong?

Where?

Why?

How do I fix this?

Real World

Cypress v3.0.0

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

...

Dashboard web app

Cypress v3 test runner sending data to API

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

Completely different sequence of calls from the test runner Cypress v3

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

Mixture of old and new data in DB

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

What data is crossing these borders?

Ohh, it is in a wiki 😖

tip: manually updating wiki examples does not ⚖️

Using Swagger or Raml?

  • API already exists
  • Other domain objects
  • Not invented here

Why not GraphQL?

  • Not just the API

Why not TypeScript?

  • Existing projects, code is crossing the boundary via JSON
{
  "title": "PostTodoRequest",
  "type": "object",
  "properties": {
    "text": {
      "type": "string"
    },
    "done": {
      "type": "boolean"
    }
  },
  "required": [
    "text",
    "done"
  ],
  "additionalProperties": false
}

json-schema

+ descriptions

{
  "title": "PostTodoRequest",
  "type": "object",
  "description": "Todo item sent by the client",
  "properties": {
    "text": {
      "type": "string",
      "description": "Todo text"
    },
    "done": {
      "type": "boolean",
      "description": "Is this todo item completed?"
    }
  },
  "required": ["text", "done"],
  "additionalProperties": false
}

json-schema

+ descriptions

+ semver

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  }
}

json-schema

+ descriptions

+ semver

+ example

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  },
  "example": {"text": "write test", "done": false}
}

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  text: 'use schemas',
  done: true,
}
assertTodoRequest(todo)
// all good

valid object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

bad object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

bad object

// Explanation

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

This for real world objects is 100% 💣

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

I'm in my happy place

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

= @cypress/schema-tools

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

some data is hard to deal with

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot() // nope, dynamic value
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok

This information is already part of the schema!!!

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
expect(
 sanitizeTodo(assertTodo(data))
).toMatchSnapshot()
exports = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "ffffffff-ffff-ffff-ffff-ffffffffffff"
}

saved snapshot

getBuilds(@user, { page: 2 })
.expect(200)
.expectResSchema("getRunResponse@1.0.0", {
   snapshot: 'second page of builds'
})

an object with 30 properties, uuids and timestamps sanitized

I am in my happy place and will never leave

JSON Schemas are Your True Testing Friend

JSON Schemas are your True Testing Friend

By Gleb Bahmutov

JSON Schemas are your True Testing Friend

In our company we have a lot of data flowing from the user’s applications (and we are backwards compatible) to the API, then this data is displayed in the web dashboard. How do we avoid accidentally breaking API contracts, while adding new features? We have started using json-schema convention and wrote a few tools around it to lock down our API protocols. Now we have full confidence in our tests, and major refactoring and releases happen without hiccups. Bonus: our tests are realistic, elegant and easy to understand. Presented at Boston JS meetup.

  • 2,931