Gleb Bahmutov, PhD

VP of Engineering, Cypress.io

JSON Schemas Update

This.JavaScript - State of Testing

About me

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

these slides

18 people. Atlanta, Philly, Boston, Chicago, NYC

Fast, easy and reliable testing for anything that runs in a browser

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

...

Dashboard web app

Cypress v1,v2 test runners sending data to API

Let's run Cypress tests in parallel!

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

Completely different sequence of calls from the test runner Cypress v3

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

Mixture of old and new data in DB

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

What data is crossing these borders?

Ohh, it is in a wiki 😖

tip: manually updating wiki examples does not ⚖️

Every time data crosses a border ...

Let's design a good solution for validating complex objects

{
  "title": "PostTodoRequest",
  "type": "object",
  "properties": {
    "text": {
      "type": "string"
    },
    "done": {
      "type": "boolean"
    }
  },
  "required": [
    "text",
    "done"
  ],
  "additionalProperties": false
}

json-schema

+ descriptions

{
  "title": "PostTodoRequest",
  "type": "object",
  "description": "Todo item sent by the client",
  "properties": {
    "text": {
      "type": "string",
      "description": "Todo text"
    },
    "done": {
      "type": "boolean",
      "description": "Is this todo item completed?"
    }
  },
  "required": ["text", "done"],
  "additionalProperties": false
}

json-schema

+ descriptions

+ semver

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  }
}

json-schema

+ descriptions

+ semver

+ example

{
  "version": {"major": 1, "minor": 0, "patch": 0},
  "schema": {
    "title": "PostTodoRequest",
    "type": "object",
    "description": "Todo item sent by the client",
    "properties": {
      "text": {
        "type": "string",
        "description": "Todo text"
      },
      "done": {
        "type": "boolean",
        "description": "Is this todo item completed?"
      }
    },
    "required": ["text", "done"],
    "additionalProperties": false
  },
  "example": {"text": "write test", "done": false}
}

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

helper curried assert function

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  text: 'use schemas',
  done: true,
}
assertTodoRequest(todo)
// all good

valid object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

invalid object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'

const assertTodoRequest = assertSchema(schemas)
    ('postTodoRequest', '1.0.0')

const todo = {
  done: true,
}
assertTodoRequest(todo)
// Error

// Explanation

invalid object

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

What went wrong?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

Why?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

How do I fix this?

json-schema

+ descriptions

+ semver

+ example

+ assert fn

Schema postTodoRequest@1.0.0 violated

Errors:
data.text is required

Current object:
{
  "done": true
}

Expected object like this:
{
  "done": false,
  "text": "do something"
}

message in the thrown error

This for real world objects is 100% 💣

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

= @cypress/schema-tools

json-schema

+ descriptions

+ semver

+ example

+ assert fn

+ document fn

  1. Make schemas

  2. Publish as NPM package

  3. Use from API and from any client

no more manual Wiki edits

API

Cypress 1.0.0

Cypress 1.1.0

Cypress 2.1.0

Cypress 3.0.0

...

Dashboard web app

database

old and new

data

Check data crossing these borders

req: PostXRequest@1.0.0

res: PostXResponse@2.1.0

JSON Schemas in

End-to-end tests

  1. Validate data fixtures

  2. Check network requests

JSON Schemas in

End-to-end tests

it('has todo fixture matching schema', () => {
  // is our fixture file correct?
  cy.fixture('todo')
    .then(api.assertSchema('PostTodoRequest', '1.0.0'))
})
it('returns new item matching schema', () => {
  cy.server()
  cy.route('POST', '/todos').as('post')
  cy.visit('/')
  cy.get('.new-todo').type('Use schemas{enter}')
  cy.wait('@post')
    .its('response.body')
    .then(api.assertSchema('PostTodoResponse', '1.0.0'))
})

JSON Schemas in

End-to-end tests: GUI + Network

some data is hard to deal with

Dynamic data

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot() // nope, dynamic value

Jest test

Cannot snapshot random UUID 😕

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok

Jest test

const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
  uuid: expect.any(String)
}) // ok

This information is already part of the schema!!!

Jest test

import { CustomFormat, CustomFormats } 
    from '@cypress/schema-tools'
const uuid: CustomFormat = {
  name: 'uuid', // the name
  description: 'GUID used through the system',
  detect: /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
  // (optional) replace actual value with this default value
  // when using to sanitize an object
  defaultValue: 'ffffffff-ffff-ffff-ffff-ffffffffffff',
}

describe custom formats in json-schema,

give it default value

schema: {
  type: 'object',
  title: 'Employee',
  properties: {
    id: {
      type: 'string',
      format: 'uuid',
    },
  },
},
example: {
  id: 'a368dbfd-08e4-4698-b9a3-b2b660a11835',
}
const data = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
expect(
 sanitizeTodo(assertTodo(data))
).toMatchSnapshot()
exports = {
  "done": false,
  "id": 2,
  "text": "do something",
  "uuid": "ffffffff-ffff-ffff-ffff-ffffffffffff"
}

saved snapshot

I'm in my happy place

JSON schemas

A portable, serializable way to describe properties of objects

JSON schemas

Useful to document and validate objects flowing through the system

Cypress REST API v2 ➡ v3 went without errors. And we still support v1 and v2 clients

JSON schemas

Can produce beautiful and helpful error messages

More information

Cypress GraphQL API

Declarative, Code-First GraphQL Schemas for JavaScript/TypeScript

Tim Griesser @tgriesser, Software Developer at Cypress.io

Gleb Bahmutov, PhD

VP of Engineering

Thank you

github.com/cypress-io/cypress ⭐️

JSON Schemas: State of Testing

By Gleb Bahmutov

JSON Schemas: State of Testing

JSON schemas is a portable declarative way to describe the shape of an object. In end-to-end testing, we can use these schemas to validate data fixtures, confirm server responses and sanitize data flowing through the application and tests for simpler assertions. In blog post form at https://www.cypress.io/blog/2018/07/10/json-schemas-are-your-true-testing-friend/

  • 356
Loading comments...

More from Gleb Bahmutov