Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
JSON Schemas are Your True Testing Friend
ConFoo.CA
these slides
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing
18 people. Atlanta, Philly, Boston, Chicago, NYC
Fast, easy and reliable testing for anything that runs in a browser
Don't trust this guy
👉
Every time code crosses a border ...
Defensive coding: checking inputs before each computation (preconditions). Sometimes checking result value before returning (postconditions).
function login(username, password) {
if (username === undefined || password === undefined) {
console.error('you forgot login or password')
// try to handle this somehow
return
}
// login ...
}Paranoid coding: checking inputs before each computation, as if the caller was evil and trying to break the function on purpose.
Just crash and explain
const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)No string concatenation - no run-time performane penalty
// 🚨
assert(condition, 'invalid tag ' + JSON.stringify(tag) +
' among ' + JSON.stringify(tags)
// ✅
la(condition, 'invalid tag', tag, 'among', tags)Just crash and explain
const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)predicate
Just crash and explain
const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)explanation
const la = require('lazy-ass')
const is = require('check-more-types')
la(is.strings(names), 'expected list of names', names)
la(is.email(loginName), 'missing email')
la(is.version(tag) || check.sha(tag), 'invalid tag', tag)explanation
🔑
STOP throwing Errors
START throwing Explanations
undefined is not a functionvs
What went wrong?
Where?
Why?
How do I fix this?
Real World
Cypress v3
$ npm install -D cypress
- Win / Mac / Linux
- MIT License
- OSS https://github.com/cypress-io/cypress
- http://on.cypress.io/introduction-to-cypress
$ npx cypress open
$ npx cypress run --record --parallel
Cypress v3.1.0
Spin N CI machines and
a single CI machine
multiple CI machines
API
Cypress 1.0.0
Cypress 1.1.0
Cypress 2.1.0
...
Dashboard web app
Cypress v3 test runner sending data to API
API
Cypress 1.0.0
Cypress 1.1.0
Cypress 2.1.0
Cypress 3.0.0
...
Dashboard web app
Completely different sequence of calls from the test runner Cypress v3
API
Cypress 1.0.0
Cypress 1.1.0
Cypress 2.1.0
Cypress 3.0.0
...
Dashboard web app
database
old and new
data
Mixture of old and new data in DB
API
Cypress 1.0.0
Cypress 1.1.0
Cypress 2.1.0
Cypress 3.0.0
...
Dashboard web app
database
old and new
data
What data is crossing these borders?
Ohh, it is in a wiki 😖
tip: manually updating wiki examples does not ⚖️
- API already exists
- Other domain objects
- Not invented here
Why not GraphQL?
- Not just the API
Why not TypeScript?
- Existing projects, code is crossing the boundary via JSON
{
"title": "PostTodoRequest",
"type": "object",
"properties": {
"text": {
"type": "string"
},
"done": {
"type": "boolean"
}
},
"required": [
"text",
"done"
],
"additionalProperties": false
}json-schema
+ descriptions
{
"title": "PostTodoRequest",
"type": "object",
"description": "Todo item sent by the client",
"properties": {
"text": {
"type": "string",
"description": "Todo text"
},
"done": {
"type": "boolean",
"description": "Is this todo item completed?"
}
},
"required": ["text", "done"],
"additionalProperties": false
}{
"version": {"major": 1, "minor": 0, "patch": 0},
"schema": {
"title": "PostTodoRequest",
"type": "object",
"description": "Todo item sent by the client",
"properties": {
"text": {
"type": "string",
"description": "Todo text"
},
"done": {
"type": "boolean",
"description": "Is this todo item completed?"
}
},
"required": ["text", "done"],
"additionalProperties": false
}
}{
"version": {"major": 1, "minor": 0, "patch": 0},
"schema": {
"title": "PostTodoRequest",
"type": "object",
"description": "Todo item sent by the client",
"properties": {
"text": {
"type": "string",
"description": "Todo text"
},
"done": {
"type": "boolean",
"description": "Is this todo item completed?"
}
},
"required": ["text", "done"],
"additionalProperties": false
},
"example": {"text": "write test", "done": false}
}import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'
const assertTodoRequest = assertSchema(schemas)
('postTodoRequest', '1.0.0')
const todo = {
text: 'use schemas',
done: true,
}
assertTodoRequest(todo)
// all goodvalid object
import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'
const assertTodoRequest = assertSchema(schemas)
('postTodoRequest', '1.0.0')
const todo = {
done: true,
}
assertTodoRequest(todo)
// Errorbad object
import { assertSchema } from '@cypress/schema-tools'
import { schemas } from '../schemas'
const assertTodoRequest = assertSchema(schemas)
('postTodoRequest', '1.0.0')
const todo = {
done: true,
}
assertTodoRequest(todo)
// Errorbad object
// Explanation
Schema postTodoRequest@1.0.0 violated
Errors:
data.text is required
Current object:
{
"done": true
}
Expected object like this:
{
"done": false,
"text": "do something"
}message in the thrown error
Schema postTodoRequest@1.0.0 violated
Errors:
data.text is required
Current object:
{
"done": true
}
Expected object like this:
{
"done": false,
"text": "do something"
}message in the thrown error
What went wrong?
Schema postTodoRequest@1.0.0 violated
Errors:
data.text is required
Current object:
{
"done": true
}
Expected object like this:
{
"done": false,
"text": "do something"
}message in the thrown error
Why?
Schema postTodoRequest@1.0.0 violated
Errors:
data.text is required
Current object:
{
"done": true
}
Expected object like this:
{
"done": false,
"text": "do something"
}message in the thrown error
How do I fix this?
Schema postTodoRequest@1.0.0 violated
Errors:
data.text is required
Current object:
{
"done": true
}
Expected object like this:
{
"done": false,
"text": "do something"
}message in the thrown error
This for real world objects is 100% 💣
Build complex schemas from simpler schemas
complex results are in private /json-schemas
const person110 = addProperty(
{
schema: person100,
description: 'Person with title',
},
{
property: 'title',
propertyType: 'string',
propertyFormat: null,
exampleValue: 'mr',
isRequired: false,
propertyDescription: 'How to address this person',
},
)Use Semver
-
Fixing a mistake - patch
-
Adding a property - minor
-
Removing a property - major
major.minor.patch
Tip: run old tests against the new schema
Use Semver
major.minor.patch
= @cypress/schema-tools
-
Make schemas
-
Publish as NPM package
-
Use from API and from client
no more manual Wiki edits
I'm in my happy place
some data is hard to deal with
Dynamic data
const data = {
"done": false,
"id": 2,
"text": "do something",
"uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot() // nope, dynamic valueconst data = {
"done": false,
"id": 2,
"text": "do something",
"uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
uuid: expect.any(String)
}) // okconst data = {
"done": false,
"id": 2,
"text": "do something",
"uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
assertTodo(data) // ok
expect(data).toMatchSnapshot({
uuid: expect.any(String)
}) // okThis information is already part of the schema!!!
import { CustomFormat, CustomFormats }
from '@cypress/schema-tools'
const uuid: CustomFormat = {
name: 'uuid', // the name
description: 'GUID used through the system',
detect: /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
// (optional) replace actual value with this default value
// when using to sanitize an object
defaultValue: 'ffffffff-ffff-ffff-ffff-ffffffffffff',
}schema: {
type: 'object',
title: 'Employee',
properties: {
id: {
type: 'string',
format: 'uuid',
},
},
},
example: {
id: 'a368dbfd-08e4-4698-b9a3-b2b660a11835',
}const data = {
"done": false,
"id": 2,
"text": "do something",
"uuid": "3372137d-b582-4e32-807d-af3021112efa"
}
expect(
sanitizeTodo(assertTodo(data))
).toMatchSnapshot()exports = {
"done": false,
"id": 2,
"text": "do something",
"uuid": "ffffffff-ffff-ffff-ffff-ffffffffffff"
}saved snapshot
getBuilds(@user, { page: 2 })
.expect(200)
.expectResSchema("getRunResponse@1.0.0", {
snapshot: 'second page of builds'
})an object with 30 properties, uuids and timestamps sanitized
Cypress REST API
r.post "/builds",
schemaVersion.set({
2: {
req: "postRunRequest@1.0.0",
res: "postRunResponse@1.0.0"
}
}),
Builds.CreateCypress REST API route
local, testing, staging environments: schema violations are thrown errors
production: schema violations are reported to Sentry
r.get "/projects/:project_slug/runs/:build_number",
loggedIn,
schemaVersion.set({
2: {
res: "getRunResponse@2.0.0"
},
3: {
res: "getRunResponse@2.1.0"
}
}),
Builds.GetRunVersioned responses
Response is trimmed to pass version X.Y.Z to avoid breaking clients
import { trim } from '@cypress/schema-tools'
const trimPerson = trim(schemas, 'Person', '1.0.0')
const person = ... // some result with lots of properties
const trimmed = trimPerson(person)
// trimmed should be valid Person 1.0.0 object
// if the values are actually matching Person@1.0.0
// all extra properties should have been removedBe liberal in what your inputs. Be conservative in your outpu.
Cypress GraphQL API
Declarative, Code-First GraphQL Schemas for JavaScript/TypeScript
Tim Griesser, Software Developer at Cypress.io
I am in my happy place and will never leave
JSON Schemas in
End-to-end tests
it('has todo fixture matching schema', () => {
// is our fixture file correct?
cy.fixture('todo')
.then(api.assertSchema('PostTodoRequest', '1.0.0'))
})it('returns new item matching schema', () => {
cy.server()
cy.route('POST', '/todos').as('post')
cy.visit('/')
cy.get('.new-todo').type('Use schemas{enter}')
cy.wait('@post')
.its('response.body')
.then(api.assertSchema('PostTodoResponse', '1.0.0'))
})JSON Schemas in
End-to-end tests: GUI + Network
JSON schemas
A portable, serializable way to describe properties of objects
JSON schemas
Useful to document and validate objects flowing through the system
Cypress REST API v2 ➡ v3 went without errors. And we still support v2 client
JSON schemas
Can produce beautiful and helpful error messages
JSON Schemas are Your True Testing Friend
ConFoo.CA
JSON Schemas are your True Testing Friend - ConFoo
By Gleb Bahmutov
JSON Schemas are your True Testing Friend - ConFoo
In our company we have a lot of data flowing from the user’s applications (and we are backwards compatible) to the API, then this data is displayed in the web dashboard. How do we avoid accidentally breaking API contracts, while adding new features? We have started using json-schema convention and wrote a few tools around it to lock down our API protocols. Now we have full confidence in our tests, and major refactoring and releases happen without hiccups. Bonus: our tests are realistic, elegant and easy to understand. Presented at Confoo.CA 2019.
