Fast Testing Using Cypress For Free

Gleb Bahmutov

survival is possible* but we need to act now

  • change your life
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

glebbahmutov.com/blog testing posts

Lots Of Testing

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Now up to 800 tests!!!

Why E2E Tests

when we have unit tests?

It Is

A Question

Of Scale

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

  • Small chunks of code like functions and classes
  • Front-end React / Angular / Vue / X components
  • Easy to test edge conditions
  • Web application
  • Easy to test the entire user flow

Running Lots of E2E Tests: Problems

  • Slow
  • Flake
  • Expensive

Cypress Cloud

Cypress Dashboard Cloud

cypress run --record

Cypress Dashboard Cloud

cypress run --record --parallel

Cypress Dashboard Cloud

  • test stats
  • screenshots
  • videos
  • traces

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

It is really nice, if you can afford it

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to save test artifacts

  • How to debug failed Cypress tests

for free 🎉

for free 🎉

for free 🎉

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Let's Run Tests On CI!

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      # Install npm dependencies, cache them
      # and run all Cypress tests
      - uses: cypress-io/github-action@v6

.github/workflows/ci.yml

Workflow run UI

Cypress GH Action details

Caching is important

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

# GitLab CI
test:
  stage: test
  parallel: 2
  script:
    - npx cypress run --env split=true

Works On Other CIs

# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run

cypress-split algorithm

  • find all Cypress specs

  • split into N chunks

  • take chunk k

The First CI Machine

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 2 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT_FILE: timings.json
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

.github/workflows/ci.yml

cypress-split timings algorithm

  • find all Cypress specs

  • look up timings

  • sort specs by duration from longest to shortest

  • fill N machine chunks

    • put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

the first machine

the second machine

Update timings.json nightly

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

Trigger the workflow manually

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'
name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'
  commit-updated-timings:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-22.04
    needs: split
    steps:
      - uses: actions/checkout@v4
        # pretty-print json string into a file
      - run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json
      - name: Commit changed spec timings ⏱️
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

Commit the changed timings

Reporting

Slow tests

Slow debugging

Fast tests

Slow debugging

Slow tests

Fast debugging

Fast tests

Fast debugging

🥇

beforeEach(() => {
  // clear the backend data
  cy.request('POST', '/reset', { todos: [] })
})

it('clears the completes todos', () => {
  cy.visit('/?addTodoDelay=1000')
  cy.get('.loaded')
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('li.todo').should('have.length', 1)
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('li.todo').should('have.length', 2)
  cy.get('.new-todo').type('third todo{enter}')
  cy.get('li.todo').should('have.length', 3)
  cy.log('**complete the third todo**')
  cy.get('li.todo').eq(2).find('.toggle').click()
  cy.get('li.todo').eq(2).should('have.class', 'completed')
  cy.get('li.todo').eq(0).should('not.have.class', 'completed')
  cy.get('li.todo').eq(1).should('not.have.class', 'completed')
  cy.log('**clear completed items**')
  cy.get('[data-cy="filter-completed"]').click()
  cy.location('hash').should('equal', '#/completed')
  cy.get('li.todo').should('have.length', 1)
  cy.contains('button', 'Clear completed').click()
  cy.log('**see all todos**')
  cy.get('[data-cy="filter-all"]').click()
  cy.location('hash').should('equal', '#/all')
  cy.get('li.todo').should('have.length', 2)
})
  1. add 3 todos
  2. mark 3rd todo completed
  3. clear completed todos

npx cypress open

E2E Test Artifacts

  • Console logs 📝

  • Screenshots on failures 🖼️

  • Browser videos 📼

  • Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

Server logs...

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 1
      start: npm start
      store-artifacts: true

.github/workflows/split.yml

zip for each CI machine

📺 cypress/videos/spec.cy.js.mp4

A test error on purpose

failure screenshot

Better Logs

// cypress.config.js
const logOptions = {
  printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)

Every Cypress command with its arguments

App console.log calls

Failed assertions

Awesome Logs Reports

// cypress.config.js
setupNodeEvents(on, config) {
  require('cypress-mochawesome-reporter/plugin')(on)
  require('cypress-terminal-report/src/installLogsPrinter')(on)
},
  
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
  cy.wait(50, { log: false }).then(() => {
    cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
  })
})
require('cypress-terminal-report/src/installLogsCollector')()

Save HTML report

E2E Test Artifacts

  • ✅ Console logs 📝

  • ✅ Screenshots on failures 🖼️

  • ✅ Browser videos 📼

  • ✅ Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

Code Coverage

"Full Code Coverage For Free"

name: ci
on: [push]
jobs:
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 4 # use 4 containers for E2E tests
      nComponent: 2 # use 2 containers for component tests
      start: npm start
      # merge E2E and component code coverage into a single report
      coverage: true

Merge coverage workflow job

Merged coverage summary

Merged coverage reports as test artifacts

Traces

same install as terminal reporter, then:

Except...

Cypress-the-company now disables plugins that compete with its cloud

https://cypresstips.substack.com/p/cypress-tips-november-2023

☢️☢️☢️☢️☢️☢️☢️

Summary

  • Run specs in parallel on CI 🤖🤖🤖
  • Save artifacts 💾 
    • screenshots, videos
    • awesome reports
    • traces perhaps

Resources

🎉 Fin 🎉

Gleb Bahmutov

Thank you 🙏

Fast Testing Using Cypress For Free

By Gleb Bahmutov

Fast Testing Using Cypress For Free

In this presentation, Gleb will explain how to speed up Cypress test execution locally and on CI using free open-source solutions that do not require paying for 3rd party dashboards or services. We will explore using API calls, data caching, and app actions to speed up each test. We also will see how to configure CI specs to split the entire test suite into multiple machines running in parallel. This presentation will benefit anyone who wants their end-to-end and component Cypress tests to finish quickly. Video at https://youtu.be/1idlr9IE0oU

  • 419