Gleb Bahmutov

VP of Engineering

Cypress.io

Cypress.io - the State of the Art End-to-end Testing Tool

Video available at https://www.youtube.com/watch?v=JL3QKQO80fs

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

we have to act today

ME

you

you

you

you

vote & lobby & rebel

https://citizensclimatelobby.org

https://rebellion.earth

if I can help I will

@bahmutov

gleb.bahmutov@gmail.com

If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.

Example: https://fab.earth

Take a deep breath

fight apathy before it kills us all

Testing is a drag

Testing does not pay

Testing is boring

This presentation

How I avoid testing

How I pick tests to write

How I make testing fun

Dr. Gleb Bahmutov, PhD

these slides

🦉 @bahmutov

I test because

I doubt myself

The simplest way to gain confidence: @ts-check

Lint Pyramid

Prettier

ESLint

@ts-check

Code is easier to read and understand

Actual linter: catches JS things that can go wrong

Strict(er) linter: catches JS and your type errors

You should try

https://slides.com/bahmutov/ts-without-ts

Step 1: code in JS

const add = (a, b) => a + b
add(2, 'foo')

Step 2: add comment

VSCode extension "Document This"

https://marketplace.visualstudio.com/items?itemName=joelday.docthis

Nice: IntelliSense

Step 3: @ts-check

CLI tsc check

$ npx tsc --noEmit --allowJs app.js 
app.js:9:8 - error TS2345: Argument of type '"foo"' is not assignable 
                           to parameter of type 'number'.

9 add(2, 'foo')
         ~~~~~


Found 1 error.

You are not documenting your code enough

IntelliSense in JS test files when you are using Cypress. Hovering over ".type" command

Documentation page for "cy.type" at https://on.cypress.io/type

Do we make more mistakes writing code or using it?

I think using it

Write code with confusing API once

1x developer

10x developers trip over it

-10x developer 🎉 🎊

Make your code simpler to understand and to use. No one should be Einstein to fix a small bug

Static types, linting, JSDoc comments, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples

E2E

integration

unit

Testing Pyramid △

(click down arrow to see types of tests)

Code

Test

API

Test

Web app

Test

There is only a test appropriate for the thing you doubt works

Thing is a function

const add = (a, b) => a + b
it('adds numbers', () => {
  expect(add(2, 3)).to.equal(5)
})

you write a test

Thing is a component

import { HelloState } from '../src/hello-x.jsx'
import { HelloState } from '../src/hello-x.jsx'
import React from 'react'
it('changes state', () => {
  cy.mount(<HelloState />)
  cy.contains('Hello Spider-man!')
  const stateToSet = { name: 'React' }
  cy.get(HelloState).invoke('setState', stateToSet)
  cy.get(HelloState)
    .its('state')
    .should('deep.equal', stateToSet)
  cy.contains('Hello React!')
})

you write a test

Thing is an API

it('yields result that has log messages', () => {
  cy.api({ url: '/' }, 'hello world')
  .then(({ messages }) => {
    const logs = Cypress._.filter(messages, {
      type: 'console',
      namespace: 'log'
    })
    expect(logs, '1 console.log message').to.have.length(1)
    expect(logs[0]).to.deep.include({
      type: 'console',
      namespace: 'log',
      message: 'processing GET /'
    })
  })
})

you write a test

https://github.com/bahmutov/cy-api

Thing is an API

you write a test

https://github.com/bahmutov/cy-api

Thing is a webapp

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

you write a test

Thing is a webapp

you write a test

Thing is a webapp's style

it('draws pizza correctly', function () {
  cy.percySnapshot('Empty Pizza')

  cy.enterDeliveryInformation()
  const toppings = ['Pepperoni', 'Chili', 'Onion']
  cy.pickToppings(...toppings)

  // make sure the web app has updated
  cy.contains('.pizza-summary__total-price', 'Total: $12.06')
  cy.percySnapshot(toppings.join(' - '))
})

you write a test

Thing is a webapp's style

you write a test

<style type="text/css">
  .st0{fill:#FFD8A1;}
  .st1{fill:#E8C08A;}
- .st2{fill:#FFDC71;}
+ .st2{fill:#71FF71;}
  .st3{fill:#DFBA86;}
</style>

changes crust color SVG

Thing is a webapp's style

you write a test

Thing is accessability

it('has good contrast', () => {
  cy.visit('/')
  cy.injectAxe()
  cy.checkA11y({
    runOnly: ['cat.color'],
  })
})

you write a test

https://glebbahmutov.com/blog/fix-low-contrast-with-cypress/

You can always write a test using the right tool

https://on.cypress.io/plugins

What should I test?

How long should my test be?

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

What should I test?

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

typical Todo application http://todomvc.com/

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Ohhh, we don't have a test for Feature C

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track manually

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

HARD

  • first test
  • two more tests
  • hundreds of tests!!!

Keep updating user stories and tests to keep them in sync

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

green: lines executed during the tests

red: lines NOT executed during the tests

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Code coverage from tests indirectly

measures implemented features tested

Keeping track via code coverage

⚠️ 100% Code Coverage ≠ 0🐞

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Unrealistic tests; subset of inputs

code does not implement the feature correctly

Code Coverage with @cypress/code-coverage

  1. Instrument code (YOU)

    1. Using Istanbul / nyc library

  2. Cypress does the rest

https://on.cypress.io/code-coverage

demo https://github.com/cypress-io/cypress-example-todomvc-redux

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

We have tested "add todo"

Need tests

https://github.com/cypress-io/cypress-example-todomvc-redux

Write more end-to-end tests

https://github.com/cypress-io/cypress-example-todomvc-redux

Ughh, missed it

Can we always write an E2E test?

https://github.com/cypress-io/cypress-example-todomvc-redux

What is this code doing?

Can I write an E2E test to hit this code?

Sometimes you cannot

This code should be unreachable from the user interface

https://github.com/cypress-io/cypress-example-todomvc-redux

Write a Unit Test!

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

@cypress/code-coverage plugin

We got this line from the unit test

@cypress/code-coverage plugin

Nice job, @cypress/code-coverage plugin

combines e2e and unit test coverage automatically

But wait, there is more ...

If you add Istanbul (nyc) to your Node.js server code

{
  "scripts": {
    "start": "node server",
    "start:coverage": "nyc --silent node server",
  }
}

https://github.com/cypress-io/code-coverage#instrument-backend-code

But wait, there is more ...

And add a GET coverage route

if (global.__coverage__) {
  // Express / Hapi / HTTP / Next.js available
  require('@cypress/code-coverage/middleware/express')(app)
}

https://github.com/cypress-io/code-coverage#instrument-backend-code

Full stack code coverage

https://github.com/cypress-io/code-coverage#instrument-backend-code

Test Coverage: The Future

State Model Coverage

it('player X wins', () => {
  play(winnerXPath)
  cy.contains('h2', 'Player X wins!')
})

it('player O wins', () => {
  play(winnerOPath)
  cy.contains('h2', 'Player O wins!')
})

draws.forEach((draw, k) => {
  it(`plays to a draw ${k}`, () => {
    const drawPath = shortestValuePaths[draw]
    play(drawPath)
    cy.contains('h2', 'Draw')
  })
})

tests = f (state machine)

https://github.com/bahmutov/tic-tac-toe

Cypress running autogenerated tests

Øredev 2018  https://vimeo.com/311931793

Use Code Coverage

  1. Guide test writing
  2. Stop build if coverage drops below X%
  3. Look beyond LoC coverage

How long should my test be?

Most unit tests are very short

  1. Arrange

  2. Act

  3. Assert

Tests are kept short because they run in the terminal. Their shortness helps to debug a failed test.

Write longer tests

Go ahead

Meaningful Long Test

https://github.com/cypress-io/cypress-example-realworld

When a test runs for too long...

Multi-page form example

https://github.com/bahmutov/cypress-example-forms

The test is too long

  1. Split into 3 tests

  2. End each test with a "checkpoint"

  3. Starts next test from a "checkpoint"

https://www.cypress.io/blog/2019/10/29/split-a-very-long-cypress-test-into-shorter-ones-using-app-actions/

class MasterForm extends React.Component {
  constructor (props) {
    super(props)
    if (window.Cypress) {
      window.app = this
    }
  }
  ...
}

Inside your app code

Now tests can control the application directly

https://www.cypress.io/blog/2019/01/03/stop-using-page-objects-and-start-using-app-actions/

expose app reference

cy.contains('Next').click()

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
  .its('app.state')
  .should('deep.equal', startOfSecondPageState)

End of the first test

cy.window()
  .its('app')
  .invoke('setState', startOfSecondPageState)

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)

Start of the second test

Start a test right from the state at the end of the previous test

Talk summary

Document, lint and test your code

Code coverage is a tool, not a goal

Do not accept slow tests

🦉 @bahmutov

Cypress.io – the State of the Art End-to-end Testing Tool

Shoot for the ✨

@bahmutov     @cypress_io

https://slides.com/bahmutov/state-of-the-art

Thank you 👏