Fast Testing Using Cypress For Free 💸💸💸

Gleb Bahmutov

gleb.dev

https://slides.com/bahmutov/fast-testing-using-cypress-for-free

📺 video at https://youtu.be/1idlr9IE0oU

Global temperature anomaly 2022 https://climate.nasa.gov/vital-signs/global-temperature/

survival is possible* but we need to act now

change your life
join an organization

@bahmutov

rebellion.global www.wedonthavetime.org

www.sierraclub.org 350.org www.greenpeace.org

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

glebbahmutov.com/blog testing posts

Lots Of Testing

https://slides.com/bahmutov/decks/mercari

https://cypress.tips/courses

gleb.dev

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Now up to 800 tests!!!

gleb.dev

Running Lots of E2E Tests: Problems

Slow
Flake
Expensive

Cypress Cloud

gleb.dev

Cypress Dashboard Cloud

cypress run --record

Cypress Dashboard Cloud

cypress run --record --parallel

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

This Presentation

How to run Cypress E2E specs in parallel
How to save test artifacts
How to debug failed Cypress tests

for free 🎉

gleb.dev

Example repo: https://github.com/bahmutov/fast-free

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Let's Run Tests On CI!

https://github.com/cypress-io/github-action

https://glebbahmutov.com/blog/trying-github-actions/

https://slides.com/bahmutov/gha-are-awesome

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      # Install npm dependencies, cache them
      # and run all Cypress tests
      - uses: cypress-io/github-action@v6

.github/workflows/ci.yml

gleb.dev

Workflow run UI

Cypress GH Action details

Caching is important

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

gleb.dev

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

https://github.com/bahmutov/cypress-split

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

https://github.com/bahmutov/cypress-split

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

cypress-split algorithm

find all Cypress specs
split into N chunks
take chunk k

https://github.com/bahmutov/find-cypress-specs

The First CI Machine

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

gleb.dev

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

gleb.dev

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 2 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

gleb.dev

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT_FILE: timings.json
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

.github/workflows/ci.yml

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

cypress-split timings algorithm

find all Cypress specs
look up timings
sort specs by duration from longest to shortest
fill N machine chunks
- put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

the first machine

the second machine

Update timings.json nightly

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

Trigger the workflow manually

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

gleb.dev

https://github.com/bahmutov/cypress-workflows

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'
  commit-updated-timings:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-22.04
    needs: split
    steps:
      - uses: actions/checkout@v4
        # pretty-print json string into a file
      - run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json
      - name: Commit changed spec timings ⏱️
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

Commit the changed timings

Reporting

Slow tests

Slow debugging

Fast tests

Slow debugging

Slow tests

Fast debugging

Fast tests

Fast debugging

gleb.dev

🥇

Example repo: https://github.com/bahmutov/test-artifacts

beforeEach(() => {
  // clear the backend data
  cy.request('POST', '/reset', { todos: [] })
})

it('clears the completes todos', () => {
  cy.visit('/?addTodoDelay=1000')
  cy.get('.loaded')
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('li.todo').should('have.length', 1)
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('li.todo').should('have.length', 2)
  cy.get('.new-todo').type('third todo{enter}')
  cy.get('li.todo').should('have.length', 3)
  cy.log('**complete the third todo**')
  cy.get('li.todo').eq(2).find('.toggle').click()
  cy.get('li.todo').eq(2).should('have.class', 'completed')
  cy.get('li.todo').eq(0).should('not.have.class', 'completed')
  cy.get('li.todo').eq(1).should('not.have.class', 'completed')
  cy.log('**clear completed items**')
  cy.get('[data-cy="filter-completed"]').click()
  cy.location('hash').should('equal', '#/completed')
  cy.get('li.todo').should('have.length', 1)
  cy.contains('button', 'Clear completed').click()
  cy.log('**see all todos**')
  cy.get('[data-cy="filter-all"]').click()
  cy.location('hash').should('equal', '#/all')
  cy.get('li.todo').should('have.length', 2)
})

add 3 todos
mark 3rd todo completed
clear completed todos

Example repo: https://github.com/bahmutov/test-artifacts

npx cypress open

E2E Test Artifacts

Console logs 📝
Screenshots on failures 🖼️
Browser videos 📼
Test commands logs 🖨️
Code coverage 🗺️
Traces 👀

gleb.dev

Server logs...

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 1
      start: npm start
      store-artifacts: true

.github/workflows/split.yml

zip for each CI machine

📺 cypress/videos/spec.cy.js.mp4

A test error on purpose

failure screenshot

Better Logs

https://github.com/archfz/cypress-terminal-report

// cypress.config.js
const logOptions = {
  printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)

https://github.com/archfz/cypress-terminal-report

Every Cypress command with its arguments

App console.log calls

Failed assertions

Awesome Logs Reports

https://github.com/archfz/cypress-terminal-report + https://github.com/LironEr/cypress-mochawesome-reporter

// cypress.config.js
setupNodeEvents(on, config) {
  require('cypress-mochawesome-reporter/plugin')(on)
  require('cypress-terminal-report/src/installLogsPrinter')(on)
},
  
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
  cy.wait(50, { log: false }).then(() => {
    cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
  })
})
require('cypress-terminal-report/src/installLogsCollector')()

Save HTML report

E2E Test Artifacts

✅ Console logs 📝
✅ Screenshots on failures 🖼️
✅ Browser videos 📼
✅ Test commands logs 🖨️
Code coverage 🗺️
Traces 👀

gleb.dev

Code Coverage

https://glebbahmutov.com/blog/full-cy-code-coverage/

"Full Code Coverage For Free"

name: ci
on: [push]
jobs:
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 4 # use 4 containers for E2E tests
      nComponent: 2 # use 2 containers for component tests
      start: npm start
      # merge E2E and component code coverage into a single report
      coverage: true

gleb.dev