Fast and Effective ... End-to-End Tests?!

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

we have to act today

ME

  • home on green electricity
  • ebike everywhere
  • ➡ online conferences instead of flying ⬅

we have to act today

US

if I can help I will

@bahmutov

gleb.bahmutov@gmail.com

If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.

Dr. Gleb Bahmutov, PhD

these slides

🦉 @bahmutov

Let's test it!

https://github.com/bahmutov/react-todo-with-hooks

$ npm i -D cypress
describe('Todo App', () => {
  it('completes an item', () => {
    // base url is stored in 
    // "cypress.json" file
    cy.visit('/')
    // there are several existing todos
    cy.get('.todo').should('have.length', 3)
  })
})

cypress/integration/todo-spec.js

Our goal: remove any obstacle to fast and easy testing

Obstacle: need to start the app then open Cypress

{
  "devDependencies": {
    "cypress": "4",
    "start-server-and-test": "1.10.11"
  },
  "scripts": {
    "start": "react-scripts start",
    "cy:open": "cypress open",
    "dev": "start-test 3000 cy:open"
  }
}
describe('Todo App', () => {
  it('completes an item', () => {
    // base url is stored in "cypress.json" file
    cy.visit('/')
    // there are several existing todos
    cy.get('.todo').should('have.length', 3)
  })
})


describe('Todo App', () => {
  it('completes an item', () => {
    // base url is stored in "cypress.json" file
    cy.visit('/')
    // there are several existing todos
    cy.get('.todo').should('have.length', 3)

    cy.log('**adding a todo**')
    cy.get('.input').type('write tests{enter}')
    cy.get('.todo').should('have.length', 4)
  })
})
describe('Todo App', () => {
  it('completes an item', () => {
    // base url is stored in "cypress.json" file
    cy.visit('/')
    // there are several existing todos
    cy.get('.todo').should('have.length', 3)

    cy.log('**adding a todo**')
    cy.get('.input').type('write tests{enter}')
    cy.get('.todo').should('have.length', 4)

    cy.log('**completing a todo**')
    cy.contains('.todo', 'write tests').contains('button', 'Complete').click()
    cy.contains('.todo', 'write tests')
      .should('have.css', 'text-decoration', 'line-through solid rgb(74, 74, 74)')
  })
})
describe('Todo App', () => {
  it('completes an item', () => {
    // base url is stored in "cypress.json" file
    cy.visit('/')
    // there are several existing todos
    cy.get('.todo').should('have.length', 3)

    cy.log('**adding a todo**')
    cy.get('.input').type('write tests{enter}')
    cy.get('.todo').should('have.length', 4)

    cy.log('**completing a todo**')
    cy.contains('.todo', 'write tests').contains('button', 'Complete').click()
    cy.contains('.todo', 'write tests')
      .should('have.css', 'text-decoration', 'line-through solid rgb(74, 74, 74)')

    cy.log('**removing a todo**')
    // due to quarantine, we have to delete an item
    // without completing it
    cy.contains('.todo', 'Meet friend for lunch').contains('button', 'x').click()
    cy.contains('.todo', 'Meet friend for lunch').should('not.exist')
  })
})

~ 5 minutes?

But was it effective?

{
  "devDependencies": {
    "cypress": "4",
    "start-server-and-test": "1.10.11",
    "@cypress/instrument-cra": "1.1.0"
  },
  "scripts": {
    "start": "react-scripts -r @cypress/instrument-cra start",
    "cy:open": "cypress open",
    "dev": "start-test 3000 cy:open"
  }
}
{
  "devDependencies": {
    "cypress": "4",
    "start-server-and-test": "1.10.11",
    "@cypress/instrument-cra": "1.1.0",
    "@cypress/code-coverage": "3.1.0"
  },
  "scripts": {
    "start": "react-scripts -r @cypress/instrument-cra start",
    "cy:open": "cypress open",
    "dev": "start-test 3000 cy:open"
  }
}
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('@cypress/code-coverage/task')(on, config)
  return config
}
// cypress/support/index.js
import '@cypress/code-coverage/support'

Save coverage

Save report at the end

$ open coverage/lcov-report/index.html

End-to-end tests are VERY effective at covering app's code

cy.log('**does not add empty todos**')
cy.get('.input').type('{enter}')
// still three items remain
cy.get('.todo').should('have.length', 3)

Code Coverage - your guide to tests to write

Are we using the right hammer?

App
TodoForm
Todo
index.js

End-to-End Tests

Let's test Todo component

{
  "devDependencies": {
    "cypress": "4",
    "start-server-and-test": "1.10.11",
    "@cypress/instrument-cra": "1.1.0",
    "@cypress/code-coverage": "3.1.0",
    "cypress-react-unit-test": "4"
  },
  "scripts": {
    "start": "react-scripts -r @cypress/instrument-cra start",
    "cy:open": "cypress open",
    "dev": "start-test 3000 cy:open"
  }
}
import React from "react";
import {Todo} from "./App";
import {mount} from 'cypress-react-unit-test'

describe('Todo', () => {
  it('renders with styles', () => {
    const todo = {
      text: 'test item',
      isCompleted: false
    }
    const TestTodo = () => <div className="app"><Todo todo={todo} /></div>
    mount(
      <TestTodo />,
      {
        stylesheets: [
          'https://cdnjs.cloudflare.com/ajax/libs/bulma/0.7.2/css/bulma.css'
        ],
        cssFile: 'src/App.css'
      }
    )
    cy.contains('.todo button', 'Complete') // regular Cypress API
  })
})

test Todo component

a single component mounted as a mini web application

Todo

Component Tests

Full browser

Code instrumentation and reporting included

Todo

Component Tests

App
TodoForm

Say no to shallow rendering

import React from "react";
import App from "./App";
import {mount} from 'cypress-react-unit-test'

describe('App', () => {
  beforeEach(() => {
    mount(
      <App />,
      {
        stylesheets: [
          'https://cdnjs.cloudflare.com/ajax/libs/bulma/0.7.2/css/bulma.css'
        ]
      }
    )
  })
  it('works', () => {
    cy.get('.todo').should('have.length', 3)
    cy.get('input.input').type('Test with Cypress{enter}')
    cy.get('.todo').should('have.length', 4)
      .contains('Meet friend for lunch')
      .find('[data-cy=remove]').click()


    cy.get('.todo').should('have.length', 3)
  })
})

test App component

Component Testing is Coming

It Passes Tests on My Machine

Cypress on Continuous Integration

  • git checkout
  • npm install
  • start the app
  • cypress run
  • stop the app

Cypress on Continuous Integration

  • git checkout
  • npm install
  • cache dependencies
  • start the app
  • cypress run
  • stop the app

Performance

Cypress on Continuous Integration

  • git checkout
  • restore cache
  • npm install
  • cache dependencies
  • start the app
  • cypress run
  • stop the app

Performance

version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record

workflows:
  version: 2
  build:
    jobs:
      - test

Docker image

Caching

Caching

typical CI config file

Install

run tests

maybe start app

Copy / paste / tweak until CI passes 🙁

defaults: &defaults
  working_directory: ~/app
  docker:
    - image: cypress/browsers:chrome67

version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # find compatible cache from previous build,
      # it should have same dependencies installed from package.json checksum
      - restore_cache:
          keys:
            - cache-{{ .Branch }}-{{ checksum "package.json" }}
      - run:
          name: Install Dependencies
          command: npm ci
      # run verify and then save cache.
      # this ensures that the Cypress verified status is cached too
      - run: npm run cy:verify
      # save new cache folder if needed
      - save_cache:
          key: cache-{{ .Branch }}-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      - run: npm run types
      - run: npm run stop-only
      # all other test jobs will run AFTER this build job finishes
      # to avoid reinstalling dependencies, we persist the source folder "app"
      # and the Cypress binary to workspace, which is the fastest way
      # for Circle jobs to pass files
      - persist_to_workspace:
          root: ~/
          paths:
            - app
            - .cache/Cypress

  4x-electron:
    <<: *defaults
    # tell CircleCI to execute this job on 4 machines simultaneously
    parallelism: 4
    steps:
      - attach_workspace:
          at: ~/
      - run:
          command: npm start
          background: true
      # runs Cypress test in load balancing (parallel) mode
      # and groups them in Cypress Dashboard under name "4x-electron"
      - run: npm run e2e:record -- --parallel --group $CIRCLE_JOB

workflows:
  version: 2
  # this workflow has 4 jobs to show case Cypress --parallel and --group flags
  # "build" installs NPM dependencies so other jobs don't have to
  #   └ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
  #   └ "4x-electron" job load balances all specs across 4 CI machines
  #   └ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
  build_and_test:
    jobs:
      - build
      # this group "4x-electron" will load balance all specs
      # across 4 CI machines
      - 4x-electron:
          requires:
            - build

Parallel config is ... more complicated

typical CI config file

- install + run jobs

- workspace

- parallel flags

# first, install Cypress, then run all tests (in parallel)
stages:
  - build
  - test

# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm"
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .npm
    - cache/Cypress
    - node_modules

# this job installs NPM dependencies and Cypress
install:
  image: cypress/base:10
  stage: build

  script:
    - npm ci
    - $(npm bin)/print-env CI
    - npm run cy:verify

# all jobs that actually run tests can use the same definition
.job_template: &job
  image: cypress/base:10
  stage: test
  script:
    # print CI environment variables for reference
    - $(npm bin)/print-env CI
    # start the server in the background
    - npm run start:ci &
    # run Cypress test in load balancing mode, pass id to tie jobs together
    - npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons

# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
  <<: *job
electrons-2:
  <<: *job
electrons-3:
  <<: *job
electrons-4:
  <<: *job
electrons-5:
  <<: *job
pipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
    }
  }

  stages {
    // first stage installs node dependencies and Cypress binary
    stage('build') {
      steps {
        // there a few default environment variables on Jenkins
        // on local Jenkins machine (assuming port 8080) see
        // http://localhost:8080/pipeline-syntax/globals#env
        echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
        sh 'npm ci'
        sh 'npm run cy:verify'
      }
    }

    stage('start local server') {
      steps {
        // start local server in the background
        // we will shut it down in "post" command block
        sh 'nohup npm start &'
      }
    }

    // this tage runs end-to-end tests, and each agent uses the workspace
    // from the previous stage
    stage('cypress parallel tests') {
      environment {
        // we will be recordint test results and video on Cypress dashboard
        // to record we need to set an environment variable
        // we can load the record key variable from credentials store
        // see https://jenkins.io/doc/book/using/using-credentials/
        CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
        // because parallel steps share the workspace they might race to delete
        // screenshots and videos folders. Tell Cypress not to delete these folders
        CYPRESS_trashAssetsBeforeRuns = 'false'
      }

      // https://jenkins.io/doc/book/pipeline/syntax/#parallel
      parallel {
        // start several test jobs in parallel, and they all
        // will use Cypress Dashboard to load balance any found spec files
        stage('tester A') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }

        // second tester runs the same command
        stage('tester B') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }
      }

    }
  }

  post {
    // shutdown the server running in the background
    always {
      echo 'Stopping local server'
      sh 'pkill -f http-server'
    }
  }
}
language: node_js

node_js:
  # Node 10.3+ includes npm@6 which has good "npm ci" command
  - 10.8

cache:
  # cache both npm modules and Cypress binary
  directories:
    - ~/.npm
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify

defaults: &defaults
  script:
    #   ## print all Travis environment variables for debugging
    - $(npm bin)/print-env TRAVIS
    - npm start -- --silent &
    - npm run cy:run -- --record --parallel --group $STAGE_NAME
    # after all tests finish running we need
    # to kill all background jobs (like "npm start &")
    - kill $(jobs -p) || true

jobs:
  include:
    # we have multiple jobs to execute using just a single stage
    # but we can pass group name via environment variable to Cypress test runner
    - stage: test
      env:
        - STAGE_NAME=1x-electron
      <<: *defaults
    # run tests in parallel by including several test jobs with same name variable
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults

Different CIs

Copy / paste / tweak until CI passes 😡

My Top 3 CIs Right NOW

Netlify 🥉

[[plugins]]
  # local Cypress plugin will test our site after it is built
  package = "netlify-plugin-cypress"

My Top 3 CIs Right NOW

CircleCI 🥈

version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run

My Top 3 CIs Right NOW

GitHub Actions 🥇

name: End-to-end tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1

Test App After Each Code Transformation

{
  "scripts": {
    "start": "react-scripts -r @cypress/instrument-cra start"
  }
}

Test the development code served using Webpack Dev Server

name: End-to-end tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1
        with:
          start: 'npm start'
          wait-on: 'http://localhost:3000'

Test the development code served using Webpack Dev Server

Test App After Each Code Transformation

{
  "scripts": {
    "start": "react-scripts -r @cypress/instrument-cra start",
    "build": "react-scripts build"
  }
}

Test the app again after building production bundle

name: End-to-end tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1
        with:
          start: 'npm start'
          wait-on: 'http://localhost:3000'
      - run: npm run build
      - uses: cypress-io/github-action@v1
        with:
          start: 'npx serve dist'
          wait-on: 'http://localhost:5000'
          config: baseUrl=http://localhost:5000
      

Test the app again after building production bundle

Test App After Each Code Transformation

# https://github.com/marketplace/actions/github-pages-action
- name: Deploy 🚀
  uses: peaceiris/actions-gh-pages@v3
  with:
     github_token: ${{ secrets.GITHUB_TOKEN }}
     publish_dir: ./dist

Test the app again after deploying

name: deployed
on:
  status:
    branches:
      - master
jobs:
  test-deployed-page:
    if: github.event.context == 'github/pages' && github.event.state == 'success'
    runs-on: ubuntu-latest
    env:
      CYPRESS_baseUrl: https://glebbahmutov.com/triple-tested/
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1

GitHub Action that runs on successful deploy to GitHub Pages

Test the app again after deploying

2 test runs (dev and prod builds)

smoke test deployed site

Never Deploy a Broken Site

If It Happens, Learn Quickly

Your Time Matters

Demand Better from The Tools You Use

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

👏 Thank you 👏

Fast and Effective ... End-to-End Tests?!

By Gleb Bahmutov

Fast and Effective ... End-to-End Tests?!

In this talk I will show how testing could be a way to speed up web application development rather than hinder it. I will show end-to-end testing as it should be: fast, providing quick feedback, and able to simulate hard to recreate edge cases. I will also show types of testing that greatly complement E2E testing: visual testing to prevent style regressions, API testing to test edge cases, accessing native browser objects, and component testing.

  • 2,643