JSON Schemas are Your True Testing Friend

ConFoo.CA

these slides

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

18 people. Atlanta, Philly, Boston, Chicago, NYC

Fast, easy and reliable testing for anything that runs in a browser

Don't trust this guy

👉

Every time code crosses a border ...

Defensive coding: checking inputs before each computation (preconditions). Sometimes checking result value before returning (postconditions).

function login(username, password) {
  if (username === undefined || password === undefined) {
     console.error('you forgot login or password')
     // try to handle this somehow
     return
  }
  // login ...
}
Paranoid coding: checking inputs before each computation, as if the caller was evil and trying to break the function on purpose.

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

No string concatenation - no run-time performane penalty

// 🚨
assert(condition, 'invalid tag ' + JSON.stringify(tag) +
  ' among ' + JSON.stringify(tags)
// ✅
la(condition, 'invalid tag', tag, 'among', tags)

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

predicate

Just crash and explain

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

explanation

const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)

explanation

🔑

STOP throwing Errors

START throwing Explanations

undefined is not a function

vs

What went wrong?

Where?

Why?

How do I fix this?

Real World

Cypress v3

$ npm install -D cypress
$ npx cypress open
$ npx cypress run --record --parallel

Cypress v3.1.0

Spin N CI machines and

a single CI machine

multiple CI machines

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

...

Dashboard web app

Cypress v3 test runner sending data to API

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

Completely different sequence of calls from the test runner Cypress v3

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

Mixture of old and new data in DB

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

What data is crossing these borders?

Ohh, it is in a wiki 😖

tip: manually updating wiki examples does not ⚖️

Using Swagger or Raml?

  • API already exists
  • Other domain objects
  • Not invented here

Why not GraphQL?

  • Not just the API

Why not TypeScript?

  • Existing projects, code is crossing the boundary via JSON
{
  "title": "PostTodoRequest",
  "type": "object",
  "properties": {
    "text": {
      "type": "string"
    },
    "done": {
      "type": "boolean"
    }
  },
  "required": [
    "text",
    "done"
  ],
  "additionalProperties": false
}

json-schema

+ descriptions

{
  "title": "PostTodoRequest",
  "type": "object",
  "description": "Todo item sent by the client",
  "properties": {
    "text": {
      "type": "string",
      "description": "Todo text"
    },
    "done": {
      "type": "boolean",
      "description": "Is this todo item completed?"
    }
  },
  "required": ["text", "done"],
  "additionalProperties": false
}

json-schema

+ descriptions

+ semver

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  }
}

json-schema

+ descriptions

+ semver

+ example

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  },
  "example": {"text": "write test", "done": false}
}

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  text: 'use schemas',
  done: true,
}
assertTodoRequest(todo)
// all good

valid object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

bad object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

bad object

// Explanation

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

What went wrong?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

Why?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

How do I fix this?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

This for real world objects is 100% 💣

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

Build complex schemas from simpler schemas

complex results are in private cypress-io/json-schemas

const person110 = addProperty(
    {
      schema: person100,
      description: 'Person with title',
    },
    {
      property: 'title',
      propertyType: 'string',
      propertyFormat: null,
      exampleValue: 'mr',
      isRequired: false,
      propertyDescription: 'How to address this person',
    },
)

Use Semver

  • Fixing a mistake - patch

  • Adding a property - minor

  • Removing a property - major

major.minor.patch

Tip: run old tests against the new schema

Use Semver

major.minor.patch

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

= @cypress/schema-tools

  1. Make schemas

  2. Publish as NPM package

  3. Use from API and from client

no more manual Wiki edits

I'm in my happy place

some data is hard to deal with

Dynamic data

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot() // nope, dynamic value
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok

This information is already part of the schema!!!

import { CustomFormat, CustomFormats } 
    from '@cypress/schema-tools'
const uuid: CustomFormat = {
  name: 'uuid', // the name
  description: 'GUID used through the system',
  detect: /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
  // (optional) replace actual value with this default value
  // when using to sanitize an object
  defaultValue: 'ffffffff-ffff-ffff-ffff-ffffffffffff',
}
schema: {
  type: 'object',
  title: 'Employee',
  properties: {
    id: {
      type: 'string',
      format: 'uuid',
    },
  },
},
example: {
  id: 'a368dbfd-08e4-4698-b9a3-b2b660a11835',
}
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
expect(
 sanitizeTodo(assertTodo(data))
).toMatchSnapshot()
exports = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "ffffffff-ffff-ffff-ffff-ffffffffffff"
}

saved snapshot

getBuilds(@user, { page: 2 })
.expect(200)
.expectResSchema("getRunResponse@1.0.0", {
   snapshot: 'second page of builds'
})

an object with 30 properties, uuids and timestamps sanitized

Cypress REST API

r.post "/builds",
  schemaVersion.set({
    2: {
      req: "postRunRequest@1.0.0",
      res: "postRunResponse@1.0.0"
    }
  }),
  Builds.Create

Cypress REST API route

local, testing, staging environments: schema violations are thrown errors

production: schema violations are reported to Sentry

r.get  "/projects/:project_slug/runs/:build_number",
        loggedIn,
        schemaVersion.set({
          2: {
            res: "getRunResponse@2.0.0"
          },
          3: {
            res: "getRunResponse@2.1.0"
          }
        }),
        Builds.GetRun

Versioned responses

Response is trimmed to pass version X.Y.Z to avoid breaking clients

import { trim } from '@cypress/schema-tools'
const trimPerson = trim(schemas, 'Person', '1.0.0')
const person = ... // some result with lots of properties
const trimmed = trimPerson(person)

// trimmed should be valid Person 1.0.0 object
// if the values are actually matching Person@1.0.0
// all extra properties should have been removed

Be liberal in what your inputs. Be conservative in your outpu.

Cypress GraphQL API

Declarative, Code-First GraphQL Schemas for JavaScript/TypeScript

Tim Griesser, Software Developer at Cypress.io

I am in my happy place and will never leave

JSON Schemas in

End-to-end tests

it('has todo fixture matching schema', () => {
  // is our fixture file correct?
  cy.fixture('todo')
    .then(api.assertSchema('PostTodoRequest', '1.0.0'))
})
it('returns new item matching schema', () => {
  cy.server()
  cy.route('POST', '/todos').as('post')
  cy.visit('/')
  cy.get('.new-todo').type('Use schemas{enter}')
  cy.wait('@post')
    .its('response.body')
    .then(api.assertSchema('PostTodoResponse', '1.0.0'))
})

JSON Schemas in

End-to-end tests: GUI + Network

JSON schemas

A portable, serializable way to describe properties of objects

JSON schemas

Useful to document and validate objects flowing through the system

Cypress REST API v2 ➡ v3 went without errors. And we still support v2 client

JSON schemas

Can produce beautiful and helpful error messages

JSON Schemas are Your True Testing Friend

ConFoo.CA

JSON Schemas are your True Testing Friend - ConFoo

By Gleb Bahmutov

JSON Schemas are your True Testing Friend - ConFoo

In our company we have a lot of data flowing from the user’s applications (and we are backwards compatible) to the API, then this data is displayed in the web dashboard. How do we avoid accidentally breaking API contracts, while adding new features? We have started using json-schema convention and wrote a few tools around it to lock down our API protocols. Now we have full confidence in our tests, and major refactoring and releases happen without hiccups. Bonus: our tests are realistic, elegant and easy to understand. Presented at Confoo.CA 2019.

  • 240
Loading comments...

More from Gleb Bahmutov