Moving Around The Testing Pyramid

E2E

integration

unit

Mocha, Ava, Jest, Jasmine, QUnit, Tape

E2E

integration

unit

import add from './add'
it('adds', () => {
  expect(add(2,3)).to.equal(5)
})

E2E

integration

unit

Mocha, Ava, Jest, Jasmine, QUnit, Tape

+

Karma, JsDom, Sinon

E2E

integration

unit

import api from './api'
nock('server.com')
  .get('/foo')
  .respond(200, 42)
it('responds', () =>
  api.foo()
  .then(result =>
    expect(result).to.equal(42)
  )
)

E2E

integration

unit

Example:

React with Enzyme

Full rendering

Shallow rendering

E2E

integration

unit

Cypress, TestCafe, Selenium, Protractor

+ stubbing

E2E

integration

unit

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login')
    .click()
  cy.contains('Enter username')
    .type('joe@joe.com')
  cy.contains('Enter password')
    .type('password{enter}')
  cy.contains('Welcome Joe!')
})

E2E

integration

unit

https://twitter.com/rauchg/status/807626710350839808

E2E

integration

unit

Need:

• real browser (DOM, storage, ...)
• state cleanup between tests
• stubbing server

E2E

integration

unit

Need:

• real browser (DOM, storage, ...)
• state cleanup between tests
• stubbing server

WAIT A MINUTE!

E2E

integration

unit

Cypress

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login').click()
})

E2E

integration

unit

Cypress for Integration

import LoginComponent from './src/login'

E2E

integration

unit

import LoginComponent from './src/login'

it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})

import mount from 'cypress-<X>-unit-test'

Cypress for Integration

E2E

integration

unit

export LoginComponent

E2E

integration

unit

export LoginComponent

Full browser

E2E

integration

unit

import LoginComponent from './src/login'

it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})

import mount from 'cypress-<X>-unit-test'

Cypress for Integration

Interact, inspect, use

cypress-vue-unit-test

cypress-react-unit-test

cypress-svelte-unit-test

cypress-hyperapp-unit-test

...

E2E

integration

unit

Why are they called

cypress-<X>-unit-test ?!

Cypress for Integration

function add(a, b) {
  return a + b
}

it('adds', () => {
  expect(add(2,3)).to.equal(5)
})

function add(a, b) {
  const el = document.getElementById('result')
  el.innerText = a + b
}

it('adds', () => {
  add(2,3)
  const el = document.getElementById('result')
  expect(el.innerText).to.equal(5)
})

function add() {
  const {a, b} = 
    JSON.parse(localStorage.getItem('inputs'))
  const el = document.getElementById('result')
  el.innerText = a + b
}

it('adds', () => {
  localStorage.setItem('inputs') =
    JSON.stringify({a: 2, b: 3})
  add()
  const el = document.getElementById('result')
  expect(el.innerText).to.equal(5)
})

component

outputs

inputs

component

outputs

inputs

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

Set up

E2E

unit

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login').click()
})

E2E

unit

it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})

E2E

unit

Use same syntax, life cycle and Cypress API

🔋🔋🔋🔋🔋🔋🔋🔋🔋🔋🔋

https://twitter.com/bahmutov/status/962367892674285568

E2E

The shape of

the quality pyramid

unit

E2E

unit

Replacing some tests

E2E

crash

reporting

linters

static

types

unit

Problem: too many successful tests

✅✅✅✅✅✅✅✅✅✅✅✅✅✅✅✅

Useful Passing Test: Snapshot testing

it('adds numbers', () => {
  expect(add(1, 2)).toMatchSnapshot()
  expect(add(100, -50)).toMatchSnapshot()
})

$ cat __snapshots__/test.js.snap 
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`add adds numbers 1`] = `3`;

exports[`add adds numbers 2`] = `50`;

Saved snapshot files

it('adds numbers', () => {
  expect(add(1, 20)).toMatchSnapshot()
  expect(add(100, -50)).toMatchSnapshot()
})

✅ build #1

expect(value).toMatchSnapshot()
    
Received value does not match stored snapshot 1.
    
    - 3
    + 21

Showing ✅ and 🛑 values works for simple values

For everything else: show diff

https://github.com/so-fancy/diff-so-fancy

Object snapshots

it('compares objects', () => {
  const o = {
    inner: {
      a: 10,
      b: 20
    },
    foo: 'foo (changed)',
    newProperty: 'here'
  }
  snapshot(o)
})

Text snapshots

it('compares multi line strings', () => {
  snapshot(`
    this is a line
    and a second line (changed)
    with number 42
  `)
})

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FF, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Shows us WHAT has changed

Shows us WHAT has changed

https://blog.percy.io/tutorials-for-your-first-visual-tests-6d733a893a18

Welcome!

screenshot difference

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}

+ <div>Welcome!</div>

app.css

index.html

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}

+ <div>Welcome!</div>

app.css

index.html

DOM diff tells HOW page has changed

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}

+ <div>Welcome!</div>

app.css

index.html

Application behavior difference

Rewind back some more

Passing test

Failing test

HTML, styles, code, cookies, storage, API calls, user interaction

test timeline

successful test run

failed test run

test timeline

Test history diff explains WHY the page has changed

successful test run

failed test run

Record Everything?

Error, stack trace, screenshot diff

Video of the accident

Car telemetry

Driver's bio data

BIG

TEST

DATA

BIG

TEST

DATA

AI / ML

BIG

TEST

DATA

AI / ML

CRYPTO

Make all your passing tests useful

✅✅✅✅✅✅✅✅✅✅✅✅✅✅✅✅

Parting Thoughts

End to end testing should do the things a user would do

Parting Thoughts

End to end test tools can do pretty good job in unit testing

Parting Thoughts

Diffing is the new "it"

• Snapshots
• Screenshot diffing
• Test behavior diffing