Gleb Bahmutov

VP of Engineering

Cypress.io

Stop! Don't write another unit test, write a Cypress test!

ConFoo.CA

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

we have to act today

ME

you

you

you

you

if I can help I will

@bahmutov

gleb.bahmutov@gmail.com

If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.

Take a deep breath

fight apathy before it kills us all

Testing is a drag

Testing does not pay

Testing is boring

This presentation

How I avoid testing

How I pick tests to write

How I make testing fun

Dr. Gleb Bahmutov, PhD

these slides

πŸ¦‰ @bahmutov

I test because

I doubt myself

The simplest way to gain confidence: @ts-check

Lint Pyramid

Prettier

ESLint

@ts-check

Code is easier to read and understand

Actual linter: catches JS things that can go wrong

Strict(er) linter: catches JS and your type errors

You should try

Step 1: code in JS

const add = (a, b) => a + b
add(2, 'foo')

Step 2: add comment

Nice: IntelliSense

Step 3: @ts-check

CLI tsc check

$ npx tsc --noEmit --allowJs app.js 
app.js:9:8 - error TS2345: Argument of type '"foo"' is not assignable 
                           to parameter of type 'number'.

9 add(2, 'foo')
         ~~~~~


Found 1 error.

You are not documenting your code enough

IntelliSense in JS test files when you are using Cypress. Hovering over ".type" command

Documentation page for "cy.type" at https://on.cypress.io/type

Do we make more mistakes writing code or using it?

I think using it

Write code with confusing API once

1x developer

10 developers trip over it

-10x developer πŸŽ‰ 🎊

Make your code simpler to understand and to use. No one should be Einstein to fix a small bug

Static types, linting, JSDoc comments, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples, examples

E2E

integration

unit

Testing Pyramid β–³

(click down arrow to see types of tests)

Code

Test

API

Test

Web app

Test

There is only a test appropriate for the thing you doubt works

My favorite Matrix shot

Thing is a function

const add = (a, b) => a + b
it('adds numbers', () => {
  expect(add(2, 3)).to.equal(5)
})

you write a test

(Cypress can run it)

Thing is a component

import { HelloState } from '../src/hello-x.jsx'
import { HelloState } from '../src/hello-x.jsx'
import React from 'react'
it('changes state', () => {
  cy.mount(<HelloState />)
  cy.contains('Hello Spider-man!')
  const stateToSet = { name: 'React' }
  cy.get(HelloState).invoke('setState', stateToSet)
  cy.get(HelloState)
    .its('state')
    .should('deep.equal', stateToSet)
  cy.contains('Hello React!')
})

you write a test (Cypress can run it)

Thing is an API

it('yields result that has log messages', () => {
  cy.api({ url: '/' }, 'hello world')
  .then(({ messages }) => {
    const logs = Cypress._.filter(messages, {
      type: 'console',
      namespace: 'log'
    })
    expect(logs, '1 console.log message').to.have.length(1)
    expect(logs[0]).to.deep.include({
      type: 'console',
      namespace: 'log',
      message: 'processing GET /'
    })
  })
})

you write a test (Cypress can run it)

Thing is an API

you write a test (Cypress can run it)

Thing is a webapp

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

you write a test (Cypress can run it)

Thing is a webapp

you write a test (Cypress can run it)

Thing is a webapp's style

it('draws pizza correctly', function () {
  cy.percySnapshot('Empty Pizza')

  cy.enterDeliveryInformation()
  const toppings = ['Pepperoni', 'Chili', 'Onion']
  cy.pickToppings(...toppings)

  // make sure the web app has updated
  cy.contains('.pizza-summary__total-price', 'Total: $12.06')
  cy.percySnapshot(toppings.join(' - '))
})

you write a test (Cypress can run it)

Thing is a webapp's style

<style type="text/css">
  .st0{fill:#FFD8A1;}
  .st1{fill:#E8C08A;}
- .st2{fill:#FFDC71;}
+ .st2{fill:#71FF71;}
  .st3{fill:#DFBA86;}
</style>

changes crust color SVG

you write a test (Cypress can run it)

Thing is a webapp's style

you write a test (Cypress can run it)

Thing is accessability

it('has good contrast', () => {
  cy.visit('/')
  cy.injectAxe()
  cy.checkA11y({
    runOnly: ['cat.color'],
  })
})

you write a test (Cypress can run it)

Thing is accessability

Thing is a Node function

const fs = require('fs')
const saveFile = (text) => 
  fs.writeFileSync('file.txt', text, 'utf8')
import {saveFile} from './app'
it('saved the file', () => {
  saveFile('hello')
  expect('file.txt').to.have.text('hello')
})

you write a test (Cypress cannot run it)

You can always write a test using the right tool

What should I test?

How long should my test be?

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

What should I test?

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

typical Todo application http://todomvc.com/

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Ohhh, we don't have a test for Feature C

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track manually

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

HARD

  • first test
  • two more tests
  • hundreds of tests!!!

Keep updating user stories and tests to keep them in sync

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

green: lines executed during the tests

red: lines NOT executed during the tests

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Code coverage from tests indirectly

measures implemented features tested

Keeping track via code coverage

⚠️ 100% Code Coverage β‰  0🐞

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Unrealistic tests; subset of inputs

code does not implement the feature correctly

Code Coverage with @cypress/code-coverage

  1. Instrument code (YOU)

    1. Using Istanbul / nyc library

  2. Cypress does the rest

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

We have tested "add todo"

Need tests

Write more end-to-end tests

Ughh, missed it

Can we always write an E2E test?

What is this code doing?

Can I write an E2E test to hit this code?

Sometimes you cannot

This code should be unreachable from the user interface

Write a Unit Test!

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

@cypress/code-coverage plugin

We got this line from the unit test

@cypress/code-coverage plugin

Nice job, @cypress/code-coverage plugin

combines e2e and unit test coverage automatically

But wait, there is more ...

If you add Istanbul (nyc) to your Node.js server code

{
  "scripts": {
    "start": "node server",
    "start:coverage": "nyc --silent node server",
  }
}

But wait, there is more ...

And add a GET coverage route

if (global.__coverage__) {
  // Express / Hapi / HTTP / Next.js available
  require('@cypress/code-coverage/middleware/express')(app)
}

Full stack code coverage

Use Code Coverage

  1. Guide test writing
  2. Stop build if coverage drops below X%
  3. Look beyond LoC coverage

Viewport coverage

Is my web app working on different resolutions?

runs the same test (right) against different viewports

import { smokeTest } from './smoke'

Cypress._.each(['macbook-15', 'iphone-6'], 
  viewport => {
    it(`works on ${viewport}`, () => {
      cy.viewport(viewport)
      smokeTest()
    })
})

Viewport coverage

Viewport coverage

How long should my test be?

Most unit tests are very short

  1. Arrange

  2. Act

  3. Assert

Tests are kept short because they run in the terminal. Their shortness helps to debug a failed test.

Write longer tests

Go ahead

Meaningful Long Test

When a test runs for too long...

Multi-page form example

The test is too long

  1. Split into 3 tests

  2. End each test with a "checkpoint"

  3. Starts next test from a "checkpoint"

class MasterForm extends React.Component {
  constructor (props) {
    super(props)
    if (window.Cypress) {
      window.app = this
    }
  }
  ...
}

Inside your app code

Now tests can control the application directly

expose app reference

DOM

storage

location

cookies

Cypress tests run in the same browser

app references

tests

// test
cy.window()
  .its('app.state')
  .then(state => {
    // call methods on state
    // or check the state data
  })
cy.contains('Next').click()

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
  .its('app.state')
  .should('deep.equal', startOfSecondPageState)

End of the first test

cy.window()
  .its('app')
  .invoke('setState', startOfSecondPageState)

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)

Start of the second test

Start a test right from the state at the end of the previous test

Talk summary

Document, lint and test your code

Code coverage is a tool, not a goal

Do not accept slow tests

πŸ¦‰ @bahmutov

Stop! Don't write another unit test, write a Cypress test!

Shoot for the ✨

@bahmutovΒ  Β  Β @cypress_io

Thank you πŸ‘

Stop! Don't write another unit test, write a Cypress test!

By Gleb Bahmutov

Stop! Don't write another unit test, write a Cypress test!

This presentation looks at the variety of ways we can test a typical web application. From unit tests to end-to-end, from functional to visual testing, from happy paths to the edge cases - there are a lot of solutions available, and each new technique gives us additional confidence in our application. This presentation will teach you how to guide test writing using code coverage using Cypress open-source test runner.

  • 471
Loading comments...

More from Gleb Bahmutov