How We Run A Lot Of End-to-End Tests At Mercari US
Speaker: Gleb Bahmutov PhD
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
Join others to fight the climate crisis
Gleb Bahmutov
Sr Director of Engineering
EveryScape
MathWorks
Kensho
Cypress.io
75 → 20
2000
8 → 100
5 → 50
300 (2000)
The same CTO
Gleb, we need our stuff to work.
-
Web app
-
ReactNative mobile app
-
APIs
-
Special projects
Agenda
-
Test speed
-
Fast(er) CI feedback
-
Test tags
-
Test reporting and triage
-
Fighting the Hydra
-
Future is 🔆
If you like it,
Put a
💍test on it...
Gleb Beyonce Bahmutov
A typical Mercari US Cypress E2E test
cy.signup(seller)
cy.createListing({
name: `Macbook one ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 198,
})
cy.createListing({
name: `Macbook two ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 199,
})
visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)
Pull request template
A typical Cypress test
-
Keep the tests readable
- custom commands, utilities, plugins
- Keep the tests readable
- custom commands, utilities, plugins
-
Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc
- Keep the tests readable
- custom commands, utilities, plugins
- Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc
-
Cache created data
- users, listing
- Keep the tests readable
- custom commands, utilities, plugins
- Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc
- Cache created data
- users, listing
-
Keep a test shorter than 3 minutes
- Keep each spec shorter than 3 minutes
- Use data-driven testing
const searches = ['Wearable', 'Running shoes', 'Dolls']
it.each(searches)(
`Filters results by status for search: %s`,
(searchKeyword) => {
const url = formSearchUrl({ searchKeyword })
cy.visitProtectedPage(url)
...
})
Plus internal web application E2E tests
660 tests * 1 minute/test
≅
11 hours to run all the tests
- How to run all the tests?
- How to run the tests for PRs?
Parallelize all the things
500+ E2E tests finish in 27 minutes using 15 CI machines
# .circleci/config.yml
orbs:
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/cypress@1
- cypress/run:
name: Nightly Cypress E2E tests
requires:
- cypress/install
record: true
# split all specs across machines
parallel: true
# use N CircleCI machines to finish quickly
# we can use a higher number of machines against the main deploy
# because it has more resources compared to the preview deploys
parallelism: 15
tags: nightlyUse the CircleCI Cypress Orb
💻
💻
💻
💻
💻
💻
💻
💻
Dev Environment
💻
💻
💻
💻
💻
💻
💻
💻
🔥 Dev Environment 🔥
🔥 🔥 🔥 🔥 🔥
💻
💻
💻
💻
💻
💻
💻
💻
We have to prioritize some spec files
Work locally on a single feature spec
Push code to run E2E tests on CI
Pull Request Flow
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
"How to Keep Cypress Tests in Another Repo While Using CircleCI"
https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci
"How to Keep Cypress Tests in Another Repo While Using GitHub Actions"
https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo/
Pull Request Flow
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
new / changed
specs
Pull Request Flow
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
💻
💻
💻
Circle CI machines
PR preview environments are isolated (GOOD), but not very powerful even compared to the DEV environment (ughh)
new / changed
specs
Find the changed specs and run them first
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent)
n=$(npx find-cypress-specs --branch main --parent --count)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs}If changed specs pass, run all or some E2E tests
Custom /cypress command in the PR comment
Test Tags
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})Effective tags
@shipping, @sanity
@regression, @mobile
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})$ find-cypress-specs --tags
Tag Tests
----------- -----
@balance 4
@careers 2
@helpcenter 69
@local 19
@login 11
@messaging 8
@mobile 77
@moble 1
@offer 6
@payment 34
@profile 55
@regression 72
@sanity 24
@search 43
@sell 76
@shipping 24
@signup 12
@w9 10Effective tags
@shipping, @sanity
@regression, @mobile
@profile
@sell
@login
@shipping
@payment
@sanity
@regression
all
@...
@sanity, @regression, all
features
Trigger tests from GitHub UI
Web Repo
PR
PR deploy
Trigger
tests
automatically
API1 Repo
PR
PR deploy
Trigger
tests
manually
API2 Repo
PR
PR deploy
Service X
PR
PR deploy
YOU can run the tests
@mobile is special
{
"scripts": {
"cy:open": "cypress open",
"cy:open:mobile": "cypress open --config viewportWidth=400,viewportHeight=600,
userAgent=\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) Mobile/14E304\""
}
}We have to run @mobile tests in a separate CI job...
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."Run any changed mobile tests
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."
changed mobile specs
changed specs
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."
GoogleBot is special
const setUserAgentHeader = () => {
cy.intercept('*', (req) => {
req.headers['user-agent'] = 'Googlebot Smartphone'
})
}
it('renders the brand page', () => {
setUserAgentHeader()
// check the server-side rendered page Google sees
cy.visitProtectedPage(brandUrl)
cy.byAriaLabel('hamburger-menu')
cy.byTestId('NotificationsButton').delay(5_000)
cy.byTestId('Error500').should('not.exist')
cy.byTestId('NameBreadcrumb', 'Apple')
})Checking the server-side rendered pages the Google crawls
Flake Is Bad
Fighting flaky tests
-
Enable test retries, look at the Cypress Dashboard
Fighting flaky tests
-
Enable test retries, look at the Cypress Dashboard
-
Increase command timeouts where necessary
-
Command - assertion pattern
Fighting flaky tests
-
spy on GraphQL calls 🎉
"Directly Spying on GraphQL Calls Made By The Application" https://www.youtube.com/watch?v=XadOqS0YNJE
spy on GraphQL calls 🎉
spy on GraphQL calls 🎉
"Set GraphQL Operation Name As Custom Header And Use It In cy.intercept" https://www.youtube.com/watch?v=AcU5mkedchM deserves a lot more ❤️
Burn Changed / New Tests
Burn Changed / New Tests
CYPRESS_burn=5 npx cypress run ...Test Management
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})
// enable TestRail integration only in some workflows
// you can use CYPRESS_enableTestRail=true or use --env CLI flag
// cypress open/run --env enableTestRail=true
if (config.env.enableTestRail) {
console.log('enableTestRail is on')
await require('cypress-testrail-simple/src/plugin')(on, config)
}cypress/plugins/index.js
Skip failing tests ASAP
it.skip('C1234 ...', () => {
...
})TestRail
Automated E2E Tests
Manual E2E Tests
API / other tests
cheaper!
faster!
never tired!
Training
-
Cypress demo sessions
-
Writing example tests for any discovered errors
-
github.com/bahmutov/cypress-workshop-basics
-
cypress.tips/courses
What we did:
Feature Flags
Did we just break:
- all the tests?
- some of the tests?
- zero tests?
cy.byTestId('LoginSubmitButton')
cy.get('button[type=submit]')
Good!
How are features served?
- explicit opt-in
- percentage
Good!
Bad 🔥
# of feature flags
# of tests
- DEV serves fixed feature flags
- All experiments are opt-in and targeted by the user ID
- E2E tests use each flag by explicit opt-in
Feature flags vs Tests
- Bonus: feature flags should be retired / removed after the experiment is over
"Control LaunchDarkly From Cypress Tests"
it('shows the feature OFF', () => {
cy.visit('/')
// feature A is OFF by default
})
it('shows the feature ON', () => {
const featureFlagKey = 'featureA'
const userId = 'USER_1234'
// target the given user to receive the first variation of the feature flag
cy.setFeatureFlagForUser(
featureFlagKey,
userId,
variationIndex: 2, // ON
)
cy.visit('/')
// feature A is ON
})npm i -D cypress-ld-control
# configure the plugin in Cypress"Control LaunchDarkly From Cypress Tests"
npm i -D cypress-ld-control
# configure the plugin in Cypressit('shows the feature OFF', () => {
cy.visit('/')
// feature A is OFF by default
})
it('shows the feature ON', () => {
const featureFlagKey = 'featureA'
const userId = 'USER_1234'
// target the given user to receive the first variation of the feature flag
cy.setFeatureFlagForUser(
featureFlagKey,
userId,
variationIndex: 2, // ON
)
cy.visit('/')
// feature A is ON
})Automated Tests
❤️
Predictability
We now catch bugs the same day or even before they are merged and deployed
recent QA team meeting
The Future
- Determine the specs to run based on code coverage and the changed source files
- Cypress v10 🎉🎉🎉 component testing
- Better triage for failed tests
How We Run A Lot Of End-to-End Tests At Mercari US