End to End Testing in the Future

Created with Sketch.

Gleb Bahmutov

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

EveryScape

virtual tours

MathWorks

MatLab on the web

Kensho

finance dashboards

11 people. Atlanta, Philly, Boston, LA

Fast, easy and reliable testing for anything that runs in a browser

I have Cypress.io t-shirts (a few)

  1. Testing pyramid

  2. E2E testing: the bad parts

  3. Cypress.io demo

  4. The future of testing

E2E

integration

unit

Smallest pieces

Testing Pyramid △

Unit tests pass...

E2E

integration

unit

Component

putting things together

E2E

integration

unit

Component

Enzyme full rendering

E2E

integration

unit

Website / app

Will Klein

Let's test!

  1. Open and control real browser

Your tests

Selenium

Driver

FF, Chrome , IE

Jennifer Shehane @jennifershehane

Let's test!

Your tests

FF, Chrome , IE

slow, flaky, unreliable tests

WebDriver = JSON over HTTP

Let's test!

Your tests

FF, Chrome , IE

Let's test!

Your tests

Headless browser

It is working ... I think

Let's test!

Your tests

Chrome Puppeteer

I need more parts

GUI showing tests

Assertion library

Automatic retries

Video recording and screenshots

Mocking and stubbing

Network control

CI support

...

$ npm install -D cypress
it('opens the page', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .should('be.visible')
})
it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

Cypress demo

  • Opening a page
  • Typical test
  • Failing test
  • API and documentation
  • Recording tests
  • CI setup

To learn more:

Egghead.io Cypress Course

Free course (same Egghead.io author!)

Cypress documentation

How does Cypress make 💵

Cypress Dashboard

See, share, and fix failing tests.

Cypress Dashboard

  • Is a paid Saas complement to Cypress app
  • Perfect place to store test results, screenshots and videos
  • Helps find why and when a test started failing

Testing of the Future

Component testing

Smart snapshots

Tests as documentation

Load balancing

Testing of the Future

Component testing

Smart snapshots

Tests as documentation

Load balancing

E2E

integration

unit

E2E

integration

unit

Need:

  • real browser (DOM, storage, ...)
  • state cleanup between tests
  • stubbing server

E2E

integration

unit

Need:

  • real browser (DOM, storage, ...)
  • state cleanup between tests
  • stubbing server

WAIT A MINUTE!

E2E

integration

unit

Cypress

it('logs user in', () => {
  cy.visit('page.com')
  cy.get('#login').click()
})

E2E

integration

unit

Cypress for Integration

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

E2E

integration

unit

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

Cypress for Integration

E2E

integration

unit

import LoginComponent from './src/login'
it('logs user in', () => {
  mount(LoginComponent)
  cy.get('#login').click()
})
import mount from 'cypress-<X>-unit-test'

storybook.js.org

Cypress for Integration

Interact, inspect, use

See: Extracting React Component

by Josh Justice

https://www.youtube.com/watch?v=Sth0bGfFOMw

Use end-to-end test runner to run your component and unit tests

Testing of the Future

Component testing

Smart snapshots

Tests as documentation

Load balancing

Why ?????

What has changed?

𝚫

(change in mathematics is usually written using Greek letter "delta")

✅ build #999

🛑 build #1000

compare current value with a good value

expect(value).toMatchSnapshot()
    
Received value does not match stored snapshot 1.
    
    - 3
    + 21

Showing ✅ and 🛑 values works for simple values

For everything else: show diff

Object snapshots

it('compares objects', () => {
  const o = {
    inner: {
      a: 10,
      b: 20
    },
    foo: 'foo (changed)',
    newProperty: 'here'
  }
  snapshot(o)
})

changed value

new value

Text snapshots

it('compares multi line strings', () => {
  snapshot(`
    this is a line
    and a second line (changed)
    with number 42
  `)
})

changed line

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FF, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

it('looks the same', () => {
  expect(cy.screenshot()).toMatchSnapshot()
})
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Screenshot diffing

[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...

Shows us WHAT has changed

Shows us WHAT has changed

Welcome!

screenshot difference

DOM

difference

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

style

dom

Rewind back

Welcome!

screenshot difference

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

DOM diff tells HOW page has changed

style

dom

.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}
+ <div>Welcome!</div>

Application behavior difference

Rewind back some more

style

dom

Passing test

Failing test

HTML, styles, code, cookies, storage, API calls, user interaction

test timeline

pixel + DOM difference

API call returned different greeting text

successful test run

failed test run

test timeline

Test history diff explains WHY the page has changed

successful test run

failed test run

Record Everything?

HAR on steroids

network + DOM + custom

Error, stack trace, screenshot diff

Video of the accident

Car telemetry

Driver's bio data

Find what really happened

See: The Testing Pyramid @AssertJS

Testing of the Future

Component testing

Smart snapshots

Tests as documentation

Load balancing

Tests are for everyone

Use Case Coverage

  • User logs in
  • User changes primary email
  • User can find an item
  • ...
describe('User', () => {
  it('logs in', () => {
    cy.visit(...)
      ...
  })
  it('changes primary email', () => {
    ...
  })
  it('can find an item', () => {
    ...
  })
})

Use Case Coverage

  • User logs in
  • User changes primary email
  • User can find an item
  • ...

Tests as documentation replace estimates (guesses) with repeated demos (tests)

Testing of the Future

Component testing

Smart snapshots

Tests as documentation

Load balancing

I don't want to wait 15 minutes

I can't wait 15 minutes

Automatic Test Load Balancing

cypress run --record --parallel
version: 2
jobs:
  test:
    parallelism: 4

CircleCI v2 example

Automatic Test Load Balancing

cypress run --record --parallel

a-spec.js

b-spec.js

c-spec.js

d-spec.js

e-spec.js

f-spec.js

...

...

...

...

Done!

Automatic Test Load Balancing

  • Run specs alphabetically
  • Short specs first
  • Run previously failed specs first
  • Run specs affected by the code first
  • ...

Automatic test load balancing makes end-to-end testing fast

End to end testing was hard

If the right tool does not exist - make it

Future of testing is bright

Component testing - ✅

Smart snapshots - in planning

Tests as documentation - PoC

Load balancing - ⏰

Cypress.io

Created with Sketch.

End to End Testing in the Future

By Gleb Bahmutov

End to End Testing in the Future

Unit testing is hard and time consuming; and worse - the users and the customers do not care! They only want the features working in the production system, everything else is development overhead. If this is the case, how do we improve the web application quality? How do we catch the bugs early? How can we test the deployed system effectively? And finally, how do we get rid of Selenium? This presentation tries to answer many of these questions in a web framework agnostic way. Presented at WeAreDevs Congress 2018, video at https://www.youtube.com/watch?v=p38bIMC-YOU

  • 5,859