Global temperature anomaly 2022 https://climate.nasa.gov/vital-signs/global-temperature/
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
glebbahmutov.com/blog testing posts
A typical Mercari US Cypress E2E test
Plus internal web application E2E tests
Now up to 800 tests!!!
expect(formatTime({ seconds: 3 }))
.to.equal('00:03')
import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()
cy.visit('/')
cy.get(...).click()
Cypress Cloud
cypress run --record
cypress run --record --parallel
every it(...) executed = test result
10 tests x 1 per day = 300 test results
10 tests x 2 per day = 600 test results 🛑
It is really nice, if you can afford it
for free 🎉
for free 🎉
for free 🎉
Example repo: https://github.com/bahmutov/fast-free
// spec-a.cy.js
it('works A', () => {
cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
cy.wait(10_000)
})
3 specs: 10 seconds, 1 minute, 10 seconds
$ npx cypress run
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
# Install npm dependencies, cache them
# and run all Cypress tests
- uses: cypress-io/github-action@v6
.github/workflows/ci.yml
Workflow run UI
Cypress GH Action details
spec-b
spec-a
spec-c
Can't we use 2 CI machines?
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-c.cy.js
2 parallel containers
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-c.cy.js
automatically compute the spec lists for each machine
const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
cypressSplit(on, config)
// IMPORTANT: return the config object
return config
},
},
})
npm i -D cypress-split
cypress.config.js
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: 2
SPLIT_INDEX: 0
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: 2
SPLIT_INDEX: 1
# GitLab CI
test:
stage: test
parallel: 2
script:
- npx cypress run --env split=true
# CircleCI
parallelism: 2
command: npx cypress run --env split=true
# many other CIs
# using process OS environment variables
job1: SPLIT=2 SPLIT_INDEX=0 npx cypress run
job2: SPLIT=2 SPLIT_INDEX=1 npx cypress run
The first machine's summary
The second machine's summary
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
The specs are split the same way
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
Split specs across 2 machines
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2, 3, 4, 5]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
Split specs across 5 machines
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
Split specs across 10 machines
spec-b
spec-a
spec-c
Is this the best we can do?
💻 1
💻 2
spec-b
spec-a
spec-c
💻 1
💻 2
We know the spec-b takes a lot longer
$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run
run this command locally
{
"durations": [
{
"spec": "cypress/e2e/spec-a.cy.js",
"duration": 10073
},
{
"spec": "cypress/e2e/spec-b.cy.js",
"duration": 60090
},
{
"spec": "cypress/e2e/spec-c.cy.js",
"duration": 10075
}
]
}
commit timings.json to the source code repo
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT_FILE: timings.json
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
.github/workflows/ci.yml
💻 1
💻 2
spec-a
spec-b
spec-c
spec-b
spec-a
spec-c
spec-b
spec-a
spec-c
the first machine
the second machine
name: nightly
on:
# run this workflow every night at 3am
schedule:
- cron: '0 3 * * *'
# or when the user triggers it from GitHub Actions page
workflow_dispatch:
jobs:
e2e:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
# we don't need Cypress own runner summary
publish-summary: false
env:
SPLIT_FILE: timings.json
SPLIT: 1
SPLIT_INDEX: 0
- name: Commit changed spec timings ⏱️
if: github.ref == 'refs/heads/main'
# https://github.com/stefanzweifel/git-auto-commit-action
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Updated spec timings
branch: main
file_pattern: timings.json
.github/workflows/nightly.yml
Trigger the workflow manually
GitHub Actions integration with GitHub = 💪💪💪
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2
.github/workflows/split.yml
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2
split-file: 'timings.json'
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2
split-file: 'timings.json'
commit-updated-timings:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-22.04
needs: split
steps:
- uses: actions/checkout@v4
# pretty-print json string into a file
- run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json
- name: Commit changed spec timings ⏱️
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Updated spec timings
branch: main
file_pattern: timings.json
Commit the changed timings
Slow tests
Slow debugging
Fast tests
Slow debugging
Slow tests
Fast debugging
Fast tests
Fast debugging
🥇
Example repo: https://github.com/bahmutov/test-artifacts
beforeEach(() => {
// clear the backend data
cy.request('POST', '/reset', { todos: [] })
})
it('clears the completes todos', () => {
cy.visit('/?addTodoDelay=1000')
cy.get('.loaded')
cy.get('.new-todo').type('first todo{enter}')
cy.get('li.todo').should('have.length', 1)
cy.get('.new-todo').type('second todo{enter}')
cy.get('li.todo').should('have.length', 2)
cy.get('.new-todo').type('third todo{enter}')
cy.get('li.todo').should('have.length', 3)
cy.log('**complete the third todo**')
cy.get('li.todo').eq(2).find('.toggle').click()
cy.get('li.todo').eq(2).should('have.class', 'completed')
cy.get('li.todo').eq(0).should('not.have.class', 'completed')
cy.get('li.todo').eq(1).should('not.have.class', 'completed')
cy.log('**clear completed items**')
cy.get('[data-cy="filter-completed"]').click()
cy.location('hash').should('equal', '#/completed')
cy.get('li.todo').should('have.length', 1)
cy.contains('button', 'Clear completed').click()
cy.log('**see all todos**')
cy.get('[data-cy="filter-all"]').click()
cy.location('hash').should('equal', '#/all')
cy.get('li.todo').should('have.length', 2)
})
Example repo: https://github.com/bahmutov/test-artifacts
npx cypress open
Server logs...
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 1
start: npm start
store-artifacts: true
.github/workflows/split.yml
zip for each CI machine
📺 cypress/videos/spec.cy.js.mp4
A test error on purpose
failure screenshot
// cypress.config.js
const logOptions = {
printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)
Every Cypress command with its arguments
App console.log calls
Failed assertions
// cypress.config.js
setupNodeEvents(on, config) {
require('cypress-mochawesome-reporter/plugin')(on)
require('cypress-terminal-report/src/installLogsPrinter')(on)
},
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
cy.wait(50, { log: false }).then(() => {
cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
})
})
require('cypress-terminal-report/src/installLogsCollector')()
Save HTML report
name: ci
on: [push]
jobs:
tests:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 4 # use 4 containers for E2E tests
nComponent: 2 # use 2 containers for component tests
start: npm start
# merge E2E and component code coverage into a single report
coverage: true
Merge coverage workflow job
Merged coverage summary
Merged coverage reports as test artifacts
same install as terminal reporter, then:
Cypress-the-company now disables plugins that compete with its cloud
https://cypresstips.substack.com/p/cypress-tips-november-2023
☢️☢️☢️☢️☢️☢️☢️
https://github.com/bahmutov/cypress-split
https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/
https://github.com/bahmutov/cypress-workflows
https://github.com/archfz/cypress-terminal-report
https://github.com/LironEr/cypress-mochawesome-reporter