Testing Web Apps on Every Commit, No Excuses

Gleb Bahmutov

@bahmutov

Distinguished Engineer

Cypress.io 

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

rebellion.global          350.org

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

(these slides)

50 people. Atlanta, Philly, Boston, NYC, the world

Fast, easy and reliable testing for anything that runs in a browser

Agenda

  • Unit vs E2E vs X tests

  • Excuses

    • We don't need E2E tests

    • The tests are hard to write

    • The tests are slow

    • The tests are flaky

    • Tests are hard to run on CI

  • Conclusions

E2E

integration

unit

Testing Pyramid △

E2E

integration

unit

Smallest pieces

Testing Pyramid △

Unit tests pass...

E2E

integration

unit

Component

E2E

integration

unit

Website / API

integration

unit

Web application

  • Open real browser
  • Load actual app
  • Interact with app like a real user
  • See if it works

E2E

Pick the right test

Testing

Test type

User flow through the web app

Individual piece of code

Server backend

Visual page appearance

Pick the right test

Testing

Test type

Individual framework component

Responsive design

E2E tests with different viewports
https://on.cypress.io/viewport

Accessability

Node code

Cypress Node test runner
(stay tuned)

Opinion: Struggling to write a test? You picked the wrong kind of test

Excuse: We do not need End-to-End tests

  • we don't need tests at all right now
  • we already have unit tests

Why is all software broken?

Will Klein

Quality software behaves the way users expect it to behave

We going to need some E2E tests

E2E

integration

unit

Really important to users

Really important to developers

E2E

integration

unit

  • logical errors
  • browser quirks
  • style errors
  • database errors
  • backend errors
  • DNS errors

Excuse: starting E2E testing is hard

  • installing special tools...
$ npm install -D cypress
// ui-spec.js
it('loads the app', () => {
  cy.visit('http://localhost:3000')
  cy.get('.todoapp').should('be.visible')
})

Mocha BDD syntax

Chai assertions

Excuse: learning testing is hard and time-consuming

  • what should I test?
  • how do I test X?
  • how to debug tests?
  • what is the best practice for Y?

Every Cypress Test Runner Pull Request

link to the documentation update

Guides

Tutorials

Api

Examples

Webinars

cypress.tips

Excuse: we don't have time to write tests

  • writing tests is slow
  • writing tests takes time away from writing features = paid work

Tests!!!

Tests are scaffolding

Everyone makes mistakes

It is up to you to have the safety rope

planning

coding

deploying

staging / QA

production

E2E

E2E

Users

planning

coding

deploying

staging / QA

production

💵

E2E

E2E

Users

💵

🐞$0

💵

💵

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💵

💵

planning

coding

deploying

staging / QA

production

💵

E2E

E2E

Users

💵

🐞$0

💵

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

💰

Answer: E2E

💵

💵

💵

Excuse: tests are flaky

  • tests sometimes fail for no reason
  • need to insert sleep commands
  • the team stops looking at failed tests
cy.get('.todo-list li')     // command
  .should('have.length', 2) // assertion

Built-in command retry-ability

{
  "retries": {
    // Configure retry attempts for `cypress run`
    // Default is 0
    "runMode": 2,
    // Configure retry attempts for `cypress open`
    // Default is 0
    "openMode": 0
  }
}

Built-in test retries

cypress.json

Built-in test retries

Built-in test retries

Excuse: tests are hard and slow on CI

  • Running lots of slow E2E tests costs money
  • CI tests are flaky
  • Debugging a test failing on CI is impossible
  • "cypress open" - GUI interactive mode

  • "cypress run" - headless mode

Cypress CLI has 2 main commands

full video of the run, screenshots of every  failure

CI setup: dream

Reality 🤯

Lots of CI providers

  • GitHub Actions
  • CircleCI
  • Netlify Build
  • BuildKite
  • Semaphore
  • AppVeyor
  • GitLab
  • Codeship
  • AzureCI
  • Shippable
  • Travis
  • Jenkins*
  • TeamCity*

* self-managed

NOT WORTH IT

Every CI config file

  • git checkout
  • install dependencies and tools
  • build code
  • start app
  • test app
  • report results

Every CI config file

  • git checkout
  • install dependencies and tools
  • build code
  • start app
  • test app
  • report results

caching!!!

Every CI config file

  • git checkout
  • install dependencies and tools
  • build code
  • start app
  • test app
  • report results

caching!!!

when is it ready?

how to stop it?

Every CI config file

  • git checkout
  • install dependencies and tools
  • build code
  • start app
  • test app
  • report results

caching!!!

when is it ready?

how to stop it?

in parallel?

version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record

workflows:
  version: 2
  build:
    jobs:
      - test

Docker image

Caching

Caching

Install

run tests

maybe start app

defaults: &defaults
  working_directory: ~/app
  docker:
    - image: cypress/browsers:chrome67

version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # find compatible cache from previous build,
      # it should have same dependencies installed from package.json checksum
      - restore_cache:
          keys:
            - cache-{{ .Branch }}-{{ checksum "package.json" }}
      - run:
          name: Install Dependencies
          command: npm ci
      # run verify and then save cache.
      # this ensures that the Cypress verified status is cached too
      - run: npm run cy:verify
      # save new cache folder if needed
      - save_cache:
          key: cache-{{ .Branch }}-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      - run: npm run types
      - run: npm run stop-only
      # all other test jobs will run AFTER this build job finishes
      # to avoid reinstalling dependencies, we persist the source folder "app"
      # and the Cypress binary to workspace, which is the fastest way
      # for Circle jobs to pass files
      - persist_to_workspace:
          root: ~/
          paths:
            - app
            - .cache/Cypress

  4x-electron:
    <<: *defaults
    # tell CircleCI to execute this job on 4 machines simultaneously
    parallelism: 4
    steps:
      - attach_workspace:
          at: ~/
      - run:
          command: npm start
          background: true
      # runs Cypress test in load balancing (parallel) mode
      # and groups them in Cypress Dashboard under name "4x-electron"
      - run: npm run e2e:record -- --parallel --group $CIRCLE_JOB

workflows:
  version: 2
  # this workflow has 4 jobs to show case Cypress --parallel and --group flags
  # "build" installs NPM dependencies so other jobs don't have to
  #   └ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
  #   └ "4x-electron" job load balances all specs across 4 CI machines
  #   └ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
  build_and_test:
    jobs:
      - build
      # this group "4x-electron" will load balance all specs
      # across 4 CI machines
      - 4x-electron:
          requires:
            - build

Parallel config is ... more complicated

- install + run jobs

- workspace

- parallel flags

Typical front-end web app testing using Cypress

# all jobs that actually run tests can use the same definition
job_template: &job_template
  image: cypress/base:10
  stage: test
  script:
    # print CI environment variables for reference
    - $(npm bin)/print-env CI
    # start the server in the background
    - npm run start:ci &
    # run Cypress test in load balancing mode
    - npm run e2e:record -- --parallel --group "electrons on GitLab CI"
  artifacts:
    when: always
    paths:
      - cypress/videos/**/*.mp4
      - cypress/screenshots/**/*.png
    expire_in: 1 day

# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
  <<: *job_template
electrons-2:
  <<: *job_template
electrons-3:
  <<: *job_template
electrons-4:
  <<: *job_template
electrons-5:
  <<: *job_template

YAML tips & tricks

Copy / paste / tweak until CI passes 🙁

# first, install Cypress, then run all tests (in parallel)
stages:
  - build
  - test

# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm"
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .npm
    - cache/Cypress
    - node_modules

# this job installs NPM dependencies and Cypress
install:
  image: cypress/base:10
  stage: build

  script:
    - npm ci
    - $(npm bin)/print-env CI
    - npm run cy:verify

# all jobs that actually run tests can use the same definition
.job_template: &job
  image: cypress/base:10
  stage: test
  script:
    # print CI environment variables for reference
    - $(npm bin)/print-env CI
    # start the server in the background
    - npm run start:ci &
    # run Cypress test in load balancing mode, pass id to tie jobs together
    - npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons

# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
  <<: *job
electrons-2:
  <<: *job
electrons-3:
  <<: *job
electrons-4:
  <<: *job
electrons-5:
  <<: *job
pipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
    }
  }

  stages {
    // first stage installs node dependencies and Cypress binary
    stage('build') {
      steps {
        // there a few default environment variables on Jenkins
        // on local Jenkins machine (assuming port 8080) see
        // http://localhost:8080/pipeline-syntax/globals#env
        echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
        sh 'npm ci'
        sh 'npm run cy:verify'
      }
    }

    stage('start local server') {
      steps {
        // start local server in the background
        // we will shut it down in "post" command block
        sh 'nohup npm start &'
      }
    }

    // this tage runs end-to-end tests, and each agent uses the workspace
    // from the previous stage
    stage('cypress parallel tests') {
      environment {
        // we will be recordint test results and video on Cypress dashboard
        // to record we need to set an environment variable
        // we can load the record key variable from credentials store
        // see https://jenkins.io/doc/book/using/using-credentials/
        CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
        // because parallel steps share the workspace they might race to delete
        // screenshots and videos folders. Tell Cypress not to delete these folders
        CYPRESS_trashAssetsBeforeRuns = 'false'
      }

      // https://jenkins.io/doc/book/pipeline/syntax/#parallel
      parallel {
        // start several test jobs in parallel, and they all
        // will use Cypress Dashboard to load balance any found spec files
        stage('tester A') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }

        // second tester runs the same command
        stage('tester B') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }
      }

    }
  }

  post {
    // shutdown the server running in the background
    always {
      echo 'Stopping local server'
      sh 'pkill -f http-server'
    }
  }
}
language: node_js

node_js:
  # Node 10.3+ includes npm@6 which has good "npm ci" command
  - 10.8

cache:
  # cache both npm modules and Cypress binary
  directories:
    - ~/.npm
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify

defaults: &defaults
  script:
    #   ## print all Travis environment variables for debugging
    - $(npm bin)/print-env TRAVIS
    - npm start -- --silent &
    - npm run cy:run -- --record --parallel --group $STAGE_NAME
    # after all tests finish running we need
    # to kill all background jobs (like "npm start &")
    - kill $(jobs -p) || true

jobs:
  include:
    # we have multiple jobs to execute using just a single stage
    # but we can pass group name via environment variable to Cypress test runner
    - stage: test
      env:
        - STAGE_NAME=1x-electron
      <<: *defaults
    # run tests in parallel by including several test jobs with same name variable
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults

Different CIs

Copy / paste / tweak until CI passes 😡

Cypress-specific CI helpers

Let us, the authors of Cypress, help you run it on CI

name: ci
on: [push, pull_request]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v1

      # Install NPM dependencies, 
      # cache them correctly,
      # and run all Cypress tests
      - name: Cypress run 🧪
        uses: cypress-io/github-action@v2
        with:
          start: npm start

.github/workflows/ci.yml

Test each commit and pull request

using GitHub Actions

Conclusions

Do not skip quality; do not skip testing

Conclusions

End-to-end testing can be fun, fast, and useful

Testing Web Apps on Every Commit, No Excuses

Gleb Bahmutov

Thank you 👏

@bahmutov

Testing Web Apps on Every Commit, No Excuses

By Gleb Bahmutov

Testing Web Apps on Every Commit, No Excuses

In this presentation, Gleb will show how every commit and every pull request can run the full set of realistic end-to-end tests, ensuring the web application is going to work for the user. He will look at the modern CI setup, benefits of clean data environments, and parallelization speed-ups. Anyone looking to learn how awesome the modern automated testing pipeline can be would benefit from this presentation. Presented at BrightTALK 2021

  • 2,723