Fast E2E Testing Using Cypress For Free

Gleb Bahmutov

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

🌎 🔥 350.org 🌎 🔥 citizensclimatelobby.org 🌎 🔥

https://cypress.tips/courses

gleb.dev

Mercari Does A Lot Of Testing

https://slides.com/bahmutov/decks/mercari

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

> 1000 E2E tests

gleb.dev

Why E2E Tests

when we have unit tests?

gleb.dev

It Is

A Question

Of Scale

gleb.dev

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

  • Small chunks of code like functions and classes
  • Front-end React / Angular / Vue / X components
  • Easy to test edge conditions
  • Web application
  • Easy to test the entire user flow

gleb.dev                    from: https://slides.com/bahmutov/the-state-of-js-testing

Running Lots of E2E Tests: Problems

  • Slow
  • Flake
  • Expensive

Cypress Cloud

gleb.dev

Cypress Dashboard Cloud

cypress run --record

gleb.dev

Cypress Dashboard Cloud

cypress run --record --parallel

Cypress Dashboard Cloud

  • test stats
  • screenshots
  • videos
  • traces

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

It is really nice, if you can afford it

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to run changed specs

  • How to run tagged tests

for free 🎉

for free 🎉

for free 🎉

gleb.dev

Example repo: https://github.com/bahmutov/fast-free

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Workflow run UI (github actions)

Cypress GH Action details

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

gleb.dev

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

https://github.com/bahmutov/cypress-split

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

https://github.com/bahmutov/cypress-split

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

# GitLab CI
test:
  stage: test
  parallel: 2
  script:
    - npx cypress run --env split=true

Works On Other CIs

https://github.com/bahmutov/cypress-split#readme

# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run

cypress-split algorithm

  • find all Cypress specs

  • split into N chunks

  • take chunk k

https://github.com/bahmutov/find-cypress-specs

gleb.dev

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

gleb.dev

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

gleb.dev

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

Tip: use bahmutov/gh-build-matrix to create strategy array for N machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

gleb.dev

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

cypress-split timings algorithm

  • find all Cypress specs

  • look up timings

  • sort specs by duration from longest to shortest

  • fill N machine chunks

    • put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

Update timings.json nightly

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

gleb.dev

https://github.com/bahmutov/cypress-workflows

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'

Which Tests To Run For This Pull Request?

gleb.dev

https://glebbahmutov.com/blog/quick-changed-specs/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

What tests/specs should we run?

https://glebbahmutov.com/blog/quick-changed-specs/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

Idea: find and run specs that were changed in the branch B with respect to the parent commit M2

npx find-cypress-specs --branch main --parent
spec-a.cy.js,spec-c.cy.js

https://glebbahmutov.com/blog/quick-changed-specs/

npx cypress run --spec spec-a.cy.js,spec-c.cy.js

Run the changed specs

We can still split these specs across N machines!

If the changed specs pass, run all tests

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to run changed specs

  • How to run tagged tests

for free 🎉

for free 🎉

for free 🎉

gleb.dev

the right

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

pull request has specs and source file changes

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

<InputError
  isError={Boolean(error)}
  type={INPUT_TYPES.TEXT}
  value={lastName}
  onChange={handleLastNameChange}
  testId="lastName"
  placeholder="Last name"
  // Custom
  id="last-name"
  autoCorrect="off"
  autoCapitalize="none"
/>

a changed source file (React JSX)

test id attribute

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

// spec-b.cy.js
cy.get('[data-testid=lastName]').should('be.visible')

specs that check elements with the test id "lastName"

// spec-z.cy.js
cy.get('[data-testid=lastName]')
  .should('not.exist')
npx find-ids --test-ids lastName
spec-b.cy.js,spec-z.cy.js

Run the found specs!

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

  • Find test ids in the changed source files

  • Run specs that use these test ids

  • (Optional) run the rest of the specs

Specs based on changes:

gleb.dev

Bonus: button does not work ☹️

gleb.dev

Look up its

test id

Launch all E2E

tests that "touch"

this element

gleb.dev

Slice Tests With Tags

gleb.dev

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})

gleb.dev

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})

Effective tags

@shipping, @sanity

@regression, @mobile

gleb.dev

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})
$ find-cypress-specs --tags

Tag            Tests
-------------  -----
@admin         21   
@analytics     7    
@balance       14   
@careers       2    
@comments      33   
@coupon        21   
@cross-border  6    
@email         41   
@furniture     3    
@guest         19   
@helpcenter    99   
@home          13
...

Effective tags

@shipping, @sanity

@regression, @mobile

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

gleb.dev

@profile

@sell

@login

@shipping

@payment

@sanity

@regression

all

@...

gleb.dev

@sanity, @regression, all

features

https://glebbahmutov.com/blog/pick-tests-using-pull-request/

+ changed specs

+ changes by test ids

When a PR is opened

Every Day 3x

Mercari runs all E2E tests (in parallel using 20 machines) in 30 minutes

  • run all changed specs
  • run all specs testing the source changes (by test id)
  • run the test tags picked by dev / QA

gleb.dev

Mercai US Today

Future: pick tests by an API call

Launch all E2E tests that "touch" this API call

Slides at https://slides.com/bahmutov/fast-and-free-e2e

👏 Thank you 👏

Gleb Bahmutov

gleb.dev

Enjoy the rest of the conference

Fast E2E Testing Using Cypress For Free

By Gleb Bahmutov

Fast E2E Testing Using Cypress For Free

Unit testing is simple, yet it only checks small pieces of code. End-to-end testing of real web applications can potentially find a lot more problems across the entire stack. Yet E2E tests are delegated to the very tip of the testing pyramid - because the tools for controlling a large application inside a real browser are brittle, and the development experience is a mixed bag. But now we have new great tools for reliable browser automation: Cypress.io and Playwright. Both tools allow quickly writing hundreds of end-to-end, component, and even API tests. Now we have a problem: how do we run all these tests quickly when we work locally or on CI? In this presentation, Gleb will explain how to speed up Cypress test execution locally and on CI using free open-source solutions that do not require paying for 3rd party dashboards or services. We will explore using API calls, data caching, and app actions to speed up each test. We also will see how to configure CI specs to split the entire test suite into multiple machines running in parallel. This presentation will benefit anyone who wants their end-to-end and component Cypress tests to finish quickly.

  • 175

More from Gleb Bahmutov