Moving Around The Testing Pyramid

E2E

integration

unit

Mocha, Ava, Jest, Jasmine, QUnit, Tape

E2E

integration

unit

import add from './add'
it('adds', () => {
  expect(add(2,3)).to.equal(5)
})

E2E

integration

unit

Mocha, Ava, Jest, Jasmine, QUnit, Tape

+

Karma, JsDom, Sinon

E2E

integration

unit

import api from './api'
nock('server.com')
  .get('/foo')
  .respond(200, 42)
it('responds', () =>
  api.foo()
  .then(result =>
    expect(result).to.equal(42)
  )
)

E2E

integration

unit

Example:

React with Enzyme

Full rendering

Shallow rendering

E2E

integration

unit

Cypress, TestCafe, Selenium, Protractor

+ stubbing

E2E

integration

unit

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login')
    .click()
  cy.contains('Enter username')
    .type('joe@joe.com')
  cy.contains('Enter password')
    .type('password{enter}')
  cy.contains('Welcome Joe!')
})

E2E

integration

unit

E2E

integration

unit

Need:

  • real browser (DOM, storage, ...)
  • state cleanup between tests
  • stubbing server

E2E

integration

unit

Need:

  • real browser (DOM, storage, ...)
  • state cleanup between tests
  • stubbing server

WAIT A MINUTE!

E2E

integration

unit

Cypress

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login').click()
})

E2E

integration

unit

Cypress for Integration

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

E2E

integration

unit

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

Cypress for Integration

E2E

integration

unit

export LoginComponent

E2E

integration

unit

export LoginComponent

Full browser

E2E

integration

unit

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

storybook.js.org

Cypress for Integration

Interact, inspect, use

cypress-vue-unit-test

cypress-react-unit-test

cypress-svelte-unit-test

cypress-hyperapp-unit-test

...

E2E

integration

unit

Why are they called

cypress-<X>-unit-test ?!

Cypress for Integration

function add(a, b) {
  return a + b
}
add(a, b)

outputs

inputs

a, b

returned value

it('adds', () => {
  expect(add(2,3)).to.equal(5)
})

Unit test

function add(a, b) {
  const el = document.getElementById('result')
  el.innerText = a + b
}
add(a, b)

outputs

inputs

a, b

DOM

it('adds', () => {
  add(2,3)
  const el = document.getElementById('result')
  expect(el.innerText).to.equal(5)
})

Integration test?

function add() {
  const {a, b} = 
    JSON.parse(localStorage.getItem('inputs'))
  const el = document.getElementById('result')
  el.innerText = a + b
}
add(a, b)

outputs

inputs

localStorage

DOM

it('adds', () => {
  localStorage.setItem('inputs') =
    JSON.stringify({a: 2, b: 3})
  add()
  const el = document.getElementById('result')
  expect(el.innerText).to.equal(5)
})

Integration test?

component

outputs

inputs

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

component

outputs

inputs

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

DOM,

localStorage,

location,

HTTP,

cookies

WW, SW,

...

Unit test

Set up

Assert

Mount

E2E

unit

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login').click()
})

E2E

unit

it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})

E2E

unit

Use same syntax, life cycle and Cypress API

πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹πŸ”‹

E2E

The shape of

the quality pyramid

unit

E2E

unit

Replacing some tests

linters

static

types

Ramda, _

E2E

The remaining holes in the pyramid

crash

reporting

linters

static

types

unit

Ramda, _

Problem: too many successful tests

βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…

Useful Passing Test: Snapshot testing

Jest / Ava / snap-shot-it

E2E Snapshots in Cypress

 PASS  ./test.js
  add
    βœ“ adds numbers (7ms)
  sub
    βœ“ subs numbers (1ms)

Snapshot Summary
 β€Ί 2 snapshots written in 1 test suite.
it('adds numbers', () => {
  expect(add(1, 2)).toMatchSnapshot()
  expect(add(100, -50)).toMatchSnapshot()
})
$ cat __snapshots__/test.js.snap 
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`add adds numbers 1`] = `3`;

exports[`add adds numbers 2`] = `50`;

Saved snapshot files

expect(value).toMatchSnapshot()
    
Received value does not match stored snapshot 1.
    
    - 3
    + 21
it('adds numbers', () => {
  expect(add(1, 20)).toMatchSnapshot()
  expect(add(100, -50)).toMatchSnapshot()
})

βœ… build #1

πŸ›‘ build #1000

compares current value with "good" saved value

expect(value).toMatchSnapshot()
    
Received value does not match stored snapshot 1.
    
    - 3
    + 21

Showing βœ… and πŸ›‘ values works for simple values

For everything else: show diff

Object snapshots

it('compares objects', () => {
  const o = {
    inner: {
      a: 10,
      b: 20
    },
    foo: 'foo (changed)',
    newProperty: 'here'
  }
  snapshot(o)
})

changed value

new value

Text snapshots

it('compares multi line strings', () => {
  snapshot(`
    this is a line
    and a second line (changed)
    with number 42
  `)
})

changed line

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FF, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Shows us WHAT has changed

Shows us WHAT has changed

Welcome!

screenshot difference

DOM

difference

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

app.css

index.html

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

app.css

index.html

DOM diff tells HOW page has changed

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

app.css

index.html

Application behavior difference

Rewind back some more

Passing test

Failing test

HTML, styles, code, cookies, storage, API calls, user interaction

test timeline

pixel + DOM difference

API call returned different greeting text

successful test run

failed test run

test timeline

Test history diff explains WHYΒ the page has changed

successful test run

failed test run

Record Everything?

HAR on steroids

network + DOM + custom

Error, stack trace, screenshot diff

Video of the accident

Car telemetry

Driver's bio data

Know what happened

automatically

BIG

TEST

DATA

Makes passing tests useful

βœ…βœ…βœ…πŸ›‘βœ…βœ…πŸ‘

BIG

TEST

DATA

AI / ML

BIG

TEST

DATA

AI / ML

CRYPTO

Make all your passing tests useful

βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…βœ…

Parting Thoughts

End to end testing should do the things a user would do

Parting Thoughts

End to end test tools can do pretty good job in unit testing

Parting Thoughts

Diffing is the new "it"

  • Snapshots
  • Screenshot diffing
  • Test behavior diffing

Thank you

@cypress_io

@be_mann

@bahmutov

Come join us!Β  Β  www.cypress.io/jobs

The Shape Of Testing Pyramid

By Gleb Bahmutov

The Shape Of Testing Pyramid

Presentation for AssertJS 2018: testing pyramid, the difference in unit tests vs integration tests, using end to end testing tool to run unit tests for components, various types of diffing: snapshot testing, screenshot diffing, application behavior diffing. Video at https://www.youtube.com/watch?v=5FnalKRjpZk

  • 949
Loading comments...

More from Gleb Bahmutov