Fast Testing Using Cypress For Free

Gleb Bahmutov

survival is possible* but we need to act now

  • change your life
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

glebbahmutov.com/blog testing posts

Lots Of Testing

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Now up to 800 tests!!!

Why E2E Tests

when we have unit tests?

It Is

A Question

Of Scale

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

  • Small chunks of code like functions and classes
  • Front-end React / Angular / Vue / X components
  • Easy to test edge conditions
  • Web application
  • Easy to test the entire user flow

Running Lots of E2E Tests: Problems

  • Slow
  • Flake
  • Expensive

Cypress Cloud

Cypress Dashboard Cloud

cypress run --record

Cypress Dashboard Cloud

cypress run --record --parallel

Cypress Dashboard Cloud

  • test stats
  • screenshots
  • videos
  • traces

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

It is really nice, if you can afford it

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to save test artifacts

  • How to debug failed Cypress tests

for free 🎉

for free 🎉

for free 🎉

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Let's Run Tests On CI!

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      # Install npm dependencies, cache them
      # and run all Cypress tests
      - uses: cypress-io/github-action@v6

.github/workflows/ci.yml

Workflow run UI

Cypress GH Action details

Caching is important

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

# GitLab CI
test:
  stage: test
  parallel: 2
  script:
    - npx cypress run --env split=true

Works On Other CIs

# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run

cypress-split algorithm

  • find all Cypress specs

  • split into N chunks

  • take chunk k

The First CI Machine

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 2 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT_FILE: timings.json
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

.github/workflows/ci.yml

cypress-split timings algorithm

  • find all Cypress specs

  • look up timings

  • sort specs by duration from longest to shortest

  • fill N machine chunks

    • put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

the first machine

the second machine

Update timings.json nightly

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

Trigger the workflow manually

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'
name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'
  commit-updated-timings:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-22.04
    needs: split
    steps:
      - uses: actions/checkout@v4
        # pretty-print json string into a file
      - run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json
      - name: Commit changed spec timings ⏱️
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

Commit the changed timings

Reporting

Slow tests

Slow debugging

Fast tests

Slow debugging

Slow tests

Fast debugging

Fast tests

Fast debugging

🥇

beforeEach(() => {
  // clear the backend data
  cy.request('POST', '/reset', { todos: [] })
})

it('clears the completes todos', () => {
  cy.visit('/?addTodoDelay=1000')
  cy.get('.loaded')
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('li.todo').should('have.length', 1)
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('li.todo').should('have.length', 2)
  cy.get('.new-todo').type('third todo{enter}')
  cy.get('li.todo').should('have.length', 3)
  cy.log('**complete the third todo**')
  cy.get('li.todo').eq(2).find('.toggle').click()
  cy.get('li.todo').eq(2).should('have.class', 'completed')
  cy.get('li.todo').eq(0).should('not.have.class', 'completed')
  cy.get('li.todo').eq(1).should('not.have.class', 'completed')
  cy.log('**clear completed items**')
  cy.get('[data-cy="filter-completed"]').click()
  cy.location('hash').should('equal', '#/completed')
  cy.get('li.todo').should('have.length', 1)
  cy.contains('button', 'Clear completed').click()
  cy.log('**see all todos**')
  cy.get('[data-cy="filter-all"]').click()
  cy.location('hash').should('equal', '#/all')
  cy.get('li.todo').should('have.length', 2)
})
  1. add 3 todos
  2. mark 3rd todo completed
  3. clear completed todos

npx cypress open

E2E Test Artifacts

  • Console logs 📝

  • Screenshots on failures 🖼️

  • Browser videos 📼

  • Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

Server logs...

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 1
      start: npm start
      store-artifacts: true

.github/workflows/split.yml

zip for each CI machine

📺 cypress/videos/spec.cy.js.mp4

A test error on purpose

failure screenshot

Better Logs

// cypress.config.js
const logOptions = {
  printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)

Every Cypress command with its arguments

App console.log calls

Failed assertions

Awesome Logs Reports

// cypress.config.js
setupNodeEvents(on, config) {
  require('cypress-mochawesome-reporter/plugin')(on)
  require('cypress-terminal-report/src/installLogsPrinter')(on)
},
  
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
  cy.wait(50, { log: false }).then(() => {
    cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
  })
})
require('cypress-terminal-report/src/installLogsCollector')()

Save HTML report

E2E Test Artifacts

  • ✅ Console logs 📝

  • ✅ Screenshots on failures 🖼️

  • ✅ Browser videos 📼

  • ✅ Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

Code Coverage

"Full Code Coverage For Free"

name: ci
on: [push]
jobs:
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 4 # use 4 containers for E2E tests
      nComponent: 2 # use 2 containers for component tests
      start: npm start
      # merge E2E and component code coverage into a single report
      coverage: true

Merge coverage workflow job

Merged coverage summary

Merged coverage reports as test artifacts

Traces

same install as terminal reporter, then:

Except...

Cypress-the-company now disables plugins that compete with its cloud

https://cypresstips.substack.com/p/cypress-tips-november-2023

☢️☢️☢️☢️☢️☢️☢️

Summary

  • Run specs in parallel on CI 🤖🤖🤖
  • Save artifacts 💾 
    • screenshots, videos
    • awesome reports
    • traces perhaps

Resources

🎉 Fin 🎉

Gleb Bahmutov

Thank you 🙏