Fast E2E Testing Using Cypress For Free

Gleb Bahmutov

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

🌎 🔥 350.org 🌎 🔥 citizensclimatelobby.org 🌎 🔥

https://cypress.tips/courses

gleb.dev

Mercari Does A Lot Of Testing

https://slides.com/bahmutov/decks/mercari

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

> 1000 E2E tests

gleb.dev

Why E2E Tests

when we have unit tests?

gleb.dev

It Is

A Question

Of Scale

gleb.dev

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

  • Small chunks of code like functions and classes
  • Front-end React / Angular / Vue / X components
  • Easy to test edge conditions
  • Web application
  • Easy to test the entire user flow

gleb.dev                    from: https://slides.com/bahmutov/the-state-of-js-testing

Running Lots of E2E Tests: Problems

  • Slow
  • Flake
  • Expensive

Cypress Cloud

gleb.dev

Cypress Dashboard Cloud

cypress run --record

gleb.dev

Cypress Dashboard Cloud

cypress run --record --parallel

Cypress Dashboard Cloud

  • test stats
  • screenshots
  • videos
  • traces

every it(...) executed = test result

10 tests x 1 per day = 300 test results

10 tests x 2 per day = 600 test results 🛑

It is really nice, if you can afford it

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to run changed specs

  • How to run tagged tests

for free 🎉

for free 🎉

for free 🎉

gleb.dev

Example repo: https://github.com/bahmutov/fast-free

// spec-a.cy.js
it('works A', () => {
  cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
  cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
  cy.wait(10_000)
})

3 specs: 10 seconds, 1 minute, 10 seconds

$ npx cypress run

Workflow run UI (github actions)

Cypress GH Action details

Waiting for 1 CI machine...

spec-b

spec-a

spec-c

Can't we use 2 CI machines?

gleb.dev

Split Specs Manually

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

Split Specs Manually

2 parallel containers

cypress-split plugin

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          spec: cypress/e2e/spec-c.cy.js

automatically compute the spec lists for each machine

https://github.com/bahmutov/cypress-split

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

npm i -D cypress-split

cypress.config.js

https://github.com/bahmutov/cypress-split

name: ci
on: push
jobs:
  e2e-1:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 0
  e2e-2:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: 2
          SPLIT_INDEX: 1

GitHub Actions Workflow

# GitLab CI
test:
  stage: test
  parallel: 2
  script:
    - npx cypress run --env split=true

Works On Other CIs

https://github.com/bahmutov/cypress-split#readme

# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run

cypress-split algorithm

  • find all Cypress specs

  • split into N chunks

  • take chunk k

https://github.com/bahmutov/find-cypress-specs

gleb.dev

The first machine's summary

The second machine's summary

The GHA Job Matrix

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

gleb.dev

Matrix Jobs UI

The specs are split the same way

Want To Go Faster?

gleb.dev

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 5 machines

name: ci
on: push
jobs:
  e2e:
    runs-on: ubuntu-22.04
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

Split specs across 10 machines

Tip: use bahmutov/gh-build-matrix to create strategy array for N machines

About that split...

spec-b

spec-a

spec-c

Is this the best we can do?

💻 1

💻 2

gleb.dev

About that split...

spec-b

spec-a

spec-c

💻 1

💻 2

We know the spec-b takes a lot longer

Idea: use spec durations

$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run

run this command locally

{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10073
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 60090
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10075
    }
  ]
}

commit timings.json to the source code repo

cypress-split timings algorithm

  • find all Cypress specs

  • look up timings

  • sort specs by duration from longest to shortest

  • fill N machine chunks

    • put each spec into the chunk with smallest sum of durations

💻 1

💻 2

spec-a

spec-b

spec-c

spec-b

spec-a

spec-c

spec-b

spec-a

spec-c

Update timings.json nightly

https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/

name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT_FILE: timings.json
          SPLIT: 1
          SPLIT_INDEX: 0
      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

.github/workflows/nightly.yml

GitHub Actions integration with GitHub = 💪💪💪

Make It Simple

gleb.dev

https://github.com/bahmutov/cypress-workflows

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2

.github/workflows/split.yml

name: split
on:
  workflow_dispatch:
jobs:
  split:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      split-file: 'timings.json'

Which Tests To Run For This Pull Request?

gleb.dev

https://glebbahmutov.com/blog/quick-changed-specs/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

What tests/specs should we run?

https://glebbahmutov.com/blog/quick-changed-specs/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

Idea: find and run specs that were changed in the branch B with respect to the parent commit M2

npx find-cypress-specs --branch main --parent
spec-a.cy.js,spec-c.cy.js

https://glebbahmutov.com/blog/quick-changed-specs/

npx cypress run --spec spec-a.cy.js,spec-c.cy.js

Run the changed specs

We can still split these specs across N machines!

If the changed specs pass, run all tests

This Presentation

  • How to run Cypress E2E specs in parallel

  • How to run changed specs

  • How to run tagged tests

for free 🎉

for free 🎉

for free 🎉

gleb.dev

the right

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

M1

M2

M3

M4

M5

M6

B1

B2

B3

Pull request X

Branch B

Main

pull request has specs and source file changes

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

<InputError
  isError={Boolean(error)}
  type={INPUT_TYPES.TEXT}
  value={lastName}
  onChange={handleLastNameChange}
  testId="lastName"
  placeholder="Last name"
  // Custom
  id="last-name"
  autoCorrect="off"
  autoCapitalize="none"
/>

a changed source file (React JSX)

test id attribute

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

// spec-b.cy.js
cy.get('[data-testid=lastName]').should('be.visible')

specs that check elements with the test id "lastName"

// spec-z.cy.js
cy.get('[data-testid=lastName]')
  .should('not.exist')
npx find-ids --test-ids lastName
spec-b.cy.js,spec-z.cy.js

Run the found specs!

gleb.dev

https://glebbahmutov.com/blog/using-test-ids-to-pick-specs-to-run/

  • Find test ids in the changed source files

  • Run specs that use these test ids

  • (Optional) run the rest of the specs

Specs based on changes:

gleb.dev

Bonus: button does not work ☹️

gleb.dev

Look up its

test id

Launch all E2E

tests that "touch"

this element

gleb.dev

Slice Tests With Tags

gleb.dev

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})

gleb.dev

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})

Effective tags

@shipping, @sanity

@regression, @mobile

gleb.dev

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})
$ find-cypress-specs --tags

Tag            Tests
-------------  -----
@admin         21   
@analytics     7    
@balance       14   
@careers       2    
@comments      33   
@coupon        21   
@cross-border  6    
@email         41   
@furniture     3    
@guest         19   
@helpcenter    99   
@home          13
...

Effective tags

@shipping, @sanity

@regression, @mobile

Learn more:

https://slides.com/bahmutov/slice-and-dice

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs 

gleb.dev

@profile

@sell

@login

@shipping

@payment

@sanity

@regression

all

@...

gleb.dev

@sanity, @regression, all

features

https://glebbahmutov.com/blog/pick-tests-using-pull-request/

+ changed specs

+ changes by test ids

When a PR is opened

Every Day 3x

Mercari runs all E2E tests (in parallel using 20 machines) in 30 minutes

  • run all changed specs
  • run all specs testing the source changes (by test id)
  • run the test tags picked by dev / QA

gleb.dev

Mercai US Today

Future: pick tests by an API call

Launch all E2E tests that "touch" this API call

Slides at https://slides.com/bahmutov/fast-and-free-e2e

👏 Thank you 👏

Gleb Bahmutov

gleb.dev

Enjoy the rest of the conference