Fast Testing Using Cypress For Free 💸💸💸

Gleb Bahmutov

gleb.dev

https://slides.com/bahmutov/e2e-free

Global temperature anomaly 2022 https://climate.nasa.gov/vital-signs/global-temperature/

survival is possible* but we need to act now

  • change your life
  • join an organization

@bahmutov

rebellion.global www.wedonthavetime.org 

www.sierraclub.org 350.org www.greenpeace.org 

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

https://cypress.tips/courses

gleb.dev

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Now up to 800 tests!!!

gleb.dev

Running Lots of E2E Tests: Problems

  • Slow
  • Flake
  • Expensive

Cypress Cloud

gleb.dev

Cypress Dashboard Cloud

cypress run --record

Cypress Dashboard Cloud

cypress run --record --parallel

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to save test artifacts

  • How to debug failed Cypress tests

for free 🎉

for free 🎉

for free 🎉

gleb.dev

Example repo: https://github.com/bahmutov/fast-free

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Let's Run Tests On CI!

https://github.com/cypress-io/github-action

https://glebbahmutov.com/blog/trying-github-actions/

https://slides.com/bahmutov/gha-are-awesome

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      # Install npm dependencies, cache them
      # and run all Cypress tests
      - uses: cypress-io/github-action@v6

.github/workflows/ci.yml

gleb.dev

Workflow run UI

Cypress GH Action details

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

gleb.dev

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

https://github.com/bahmutov/cypress-split

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

https://github.com/bahmutov/cypress-split

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

# GitLab CI
test:
  stage: test
  parallel: 2
  script:
    - npx cypress run --env split=true

Works On Other CIs

https://github.com/bahmutov/cypress-split#readme

# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run

cypress-split algorithm

  • find all Cypress specs

  • split into N chunks

  • take chunk k

https://github.com/bahmutov/find-cypress-specs

The First CI Machine

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

gleb.dev

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

gleb.dev

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 2 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

gleb.dev

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT_FILE: timings.json
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

.github/workflows/ci.yml

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

cypress-split timings algorithm

  • find all Cypress specs

  • look up timings

  • sort specs by duration from longest to shortest

  • fill N machine chunks

    • put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

the first machine

the second machine

Update timings.json nightly

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

gleb.dev

https://github.com/bahmutov/cypress-workflows

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

gleb.dev

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'

gleb.dev

Reporting

Slow tests

Slow debugging

Fast tests

Slow debugging

Slow tests

Fast debugging

Fast tests

Fast debugging

gleb.dev

🥇

Example repo: https://github.com/bahmutov/test-artifacts

beforeEach(() => {
  // clear the backend data
  cy.request('POST', '/reset', { todos: [] })
})

it('clears the completes todos', () => {
  cy.visit('/?addTodoDelay=1000')
  cy.get('.loaded')
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('li.todo').should('have.length', 1)
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('li.todo').should('have.length', 2)
  cy.get('.new-todo').type('third todo{enter}')
  cy.get('li.todo').should('have.length', 3)
  cy.log('**complete the third todo**')
  cy.get('li.todo').eq(2).find('.toggle').click()
  cy.get('li.todo').eq(2).should('have.class', 'completed')
  cy.get('li.todo').eq(0).should('not.have.class', 'completed')
  cy.get('li.todo').eq(1).should('not.have.class', 'completed')
  cy.log('**clear completed items**')
  cy.get('[data-cy="filter-completed"]').click()
  cy.location('hash').should('equal', '#/completed')
  cy.get('li.todo').should('have.length', 1)
  cy.contains('button', 'Clear completed').click()
  cy.log('**see all todos**')
  cy.get('[data-cy="filter-all"]').click()
  cy.location('hash').should('equal', '#/all')
  cy.get('li.todo').should('have.length', 2)
})
  1. add 3 todos
  2. mark 3rd todo completed
  3. clear completed todos

Example repo: https://github.com/bahmutov/test-artifacts

npx cypress open

E2E Test Artifacts

  • Console logs 📝

  • Screenshots on failures 🖼️

  • Browser videos 📼

  • Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

gleb.dev

Server logs... not much

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 1
      start: npm start
      store-artifacts: true

.github/workflows/split.yml

Test Artifacts

saves cypress/screenshots and cypress/videos

zip for each CI machine

📺 cypress/videos/spec.cy.js.mp4

A test error on purpose

failure screenshot

Better Logs

https://github.com/archfz/cypress-terminal-report

// cypress.config.js
const logOptions = {
  printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)

https://github.com/archfz/cypress-terminal-report

Every Cypress command with its arguments

App console.log calls

Failed assertions

Awesome Logs Reports

https://github.com/archfz/cypress-terminal-report + https://github.com/LironEr/cypress-mochawesome-reporter

// cypress.config.js
setupNodeEvents(on, config) {
  require('cypress-mochawesome-reporter/plugin')(on)
  require('cypress-terminal-report/src/installLogsPrinter')(on)
},
  
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
  cy.wait(50, { log: false }).then(() => {
    cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
  })
})
require('cypress-terminal-report/src/installLogsCollector')()

Save HTML report

E2E Test Artifacts

  • ✅ Console logs 📝

  • ✅ Screenshots on failures 🖼️

  • ✅ Browser videos 📼

  • ✅ Test commands logs 🖨️

  • Code coverage 🗺️

  • Traces 👀

gleb.dev

Code Coverage

https://glebbahmutov.com/blog/full-cy-code-coverage/

"Full Code Coverage For Free"

name: ci
on: [push]
jobs:
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 4 # use 4 containers for E2E tests
      nComponent: 2 # use 2 containers for component tests
      start: npm start
      # merge E2E and component code coverage into a single report
      coverage: true

gleb.dev

using https://github.com/bahmutov/cypress-code-coverage plugin

Merge coverage workflow job

gleb.dev

Merged coverage summary

gleb.dev

Merged coverage reports as test artifacts

gleb.dev

Traces

https://currents.dev/posts/currents-cypress-replay-debugger

same install as terminal reporter, then:

gleb.dev

Except...

Cypress-the-company now disables plugins that compete with its cloud

https://cypresstips.substack.com/p/cypress-tips-november-2023

☢️☢️☢️☢️☢️☢️☢️

Summary

  • Run specs in parallel on CI 🤖🤖🤖
  • Save artifacts 💾 
    • screenshots, videos
    • awesome reports
    • traces perhaps

gleb.dev

Resources

https://github.com/bahmutov/cypress-split

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

https://github.com/bahmutov/cypress-workflows

https://github.com/archfz/cypress-terminal-report

https://github.com/LironEr/cypress-mochawesome-reporter

https://glebbahmutov.com/blog/full-cy-code-coverage/

https://currents.dev/posts/currents-cypress-replay-debugger

🎉 Fin 🎉

Gleb Bahmutov

gleb.dev

Thank you 🙏

https://slides.com/bahmutov/e2e-free

👀 "The Big Dig: The story of America's most expensive road"

https://www.youtube.com/playlist?list=PLMQKK3_a14M3A-SQdVVWhOfOw8xRUuueJ

For All Boston / MA

peeps

Fast Testing Using Cypress For Free

By Gleb Bahmutov

Fast Testing Using Cypress For Free

In this presentation, Gleb will explain how to speed up Cypress test execution locally and on CI using free open-source solutions that do not require paying for 3rd party dashboards or services. We will explore using API calls, data caching, and app actions to speed up each test. We also will see how to configure CI specs to split the entire test suite into multiple machines running in parallel. This presentation will benefit anyone who wants their end-to-end and component Cypress tests to finish quickly. Boston Code Camp 30 minutes

  • 877

More from Gleb Bahmutov