C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
75 → 20
2000
8 → 100
5 → 50
300 (2000)
Gleb, we need our stuff to work.
If you like it,
Put a
💍test on it...
Gleb Beyonce Bahmutov
A typical Mercari US Cypress E2E test
cy.signup(seller)
cy.createListing({
name: `Macbook one ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 198,
})
cy.createListing({
name: `Macbook two ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 199,
})
visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)
Pull request template
A typical Cypress test
const searches = ['Wearable', 'Running shoes', 'Dolls']
it.each(searches)(
`Filters results by status for search: %s`,
(searchKeyword) => {
const url = formSearchUrl({ searchKeyword })
cy.visitProtectedPage(url)
...
})
Plus internal web application E2E tests
660 tests * 1 minute/test
≅
11 hours to run all the tests
500+ E2E tests finish in 27 minutes using 15 CI machines
# .circleci/config.yml
orbs:
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/cypress@1
- cypress/run:
name: Nightly Cypress E2E tests
requires:
- cypress/install
record: true
# split all specs across machines
parallel: true
# use N CircleCI machines to finish quickly
# we can use a higher number of machines against the main deploy
# because it has more resources compared to the preview deploys
parallelism: 15
tags: nightly
Use the CircleCI Cypress Orb
💻
💻
💻
💻
💻
💻
💻
💻
Dev Environment
💻
💻
💻
💻
💻
💻
💻
💻
🔥 Dev Environment 🔥
🔥 🔥 🔥 🔥 🔥
💻
💻
💻
💻
💻
💻
💻
💻
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
"How to Keep Cypress Tests in Another Repo While Using CircleCI"
https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci
"How to Keep Cypress Tests in Another Repo While Using GitHub Actions"
https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo/
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
new / changed
specs
Web Repo
E2E Repo
PR
PR deploy
Trigger E2E tests
💻
💻
💻
Circle CI machines
PR preview environments are isolated (GOOD), but not very powerful even compared to the DEV environment (ughh)
new / changed
specs
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent)
n=$(npx find-cypress-specs --branch main --parent --count)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs}
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})
Effective tags
@shipping, @sanity
@regression, @mobile
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})
$ find-cypress-specs --tags
Tag Tests
----------- -----
@balance 4
@careers 2
@helpcenter 69
@local 19
@login 11
@messaging 8
@mobile 77
@moble 1
@offer 6
@payment 34
@profile 55
@regression 72
@sanity 24
@search 43
@sell 76
@shipping 24
@signup 12
@w9 10
Effective tags
@shipping, @sanity
@regression, @mobile
@profile
@sell
@login
@shipping
@payment
@sanity
@regression
all
@...
@sanity, @regression, all
features
Web Repo
PR
PR deploy
Trigger
tests
automatically
API1 Repo
PR
PR deploy
Trigger
tests
manually
API2 Repo
PR
PR deploy
Service X
PR
PR deploy
{
"scripts": {
"cy:open": "cypress open",
"cy:open:mobile": "cypress open --config viewportWidth=400,viewportHeight=600,
userAgent=\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) Mobile/14E304\""
}
}
We have to run @mobile tests in a separate CI job...
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."
Run any changed mobile tests
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."
changed mobile specs
changed specs
# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)
if [ ${n} -lt 1 ]; then
echo "No Cypress specs changed, exiting..."
exit 0
fi
npx cypress run --record --parallel --spec ${specs} \
--config userAgent="Mobile ...."
const setUserAgentHeader = () => {
cy.intercept('*', (req) => {
req.headers['user-agent'] = 'Googlebot Smartphone'
})
}
it('renders the brand page', () => {
setUserAgentHeader()
// check the server-side rendered page Google sees
cy.visitProtectedPage(brandUrl)
cy.byAriaLabel('hamburger-menu')
cy.byTestId('NotificationsButton').delay(5_000)
cy.byTestId('Error500').should('not.exist')
cy.byTestId('NameBreadcrumb', 'Apple')
})
Checking the server-side rendered pages the Google crawls
"Directly Spying on GraphQL Calls Made By The Application" https://www.youtube.com/watch?v=XadOqS0YNJE
"Set GraphQL Operation Name As Custom Header And Use It In cy.intercept" https://www.youtube.com/watch?v=AcU5mkedchM deserves a lot more ❤️
CYPRESS_burn=5 npx cypress run ...
describe('Shipping', { tags: '@shipping' }, () => {
it(
'C1234 uses the default Mercari shipping',
{ tags: ['@sanity', '@regression', '@mobile'] },
() => {
...
}
)
})
// enable TestRail integration only in some workflows
// you can use CYPRESS_enableTestRail=true or use --env CLI flag
// cypress open/run --env enableTestRail=true
if (config.env.enableTestRail) {
console.log('enableTestRail is on')
await require('cypress-testrail-simple/src/plugin')(on, config)
}
cypress/plugins/index.js
it.skip('C1234 ...', () => {
...
})
Automated E2E Tests
Manual E2E Tests
API / other tests
cheaper!
faster!
never tired!
Did we just break:
cy.byTestId('LoginSubmitButton')
cy.get('button[type=submit]')
Good!
How are features served?
Good!
Bad 🔥
# of feature flags
# of tests
"Control LaunchDarkly From Cypress Tests"
it('shows the feature OFF', () => {
cy.visit('/')
// feature A is OFF by default
})
it('shows the feature ON', () => {
const featureFlagKey = 'featureA'
const userId = 'USER_1234'
// target the given user to receive the first variation of the feature flag
cy.setFeatureFlagForUser(
featureFlagKey,
userId,
variationIndex: 2, // ON
)
cy.visit('/')
// feature A is ON
})
npm i -D cypress-ld-control
# configure the plugin in Cypress
"Control LaunchDarkly From Cypress Tests"
npm i -D cypress-ld-control
# configure the plugin in Cypress
it('shows the feature OFF', () => {
cy.visit('/')
// feature A is OFF by default
})
it('shows the feature ON', () => {
const featureFlagKey = 'featureA'
const userId = 'USER_1234'
// target the given user to receive the first variation of the feature flag
cy.setFeatureFlagForUser(
featureFlagKey,
userId,
variationIndex: 2, // ON
)
cy.visit('/')
// feature A is ON
})
We now catch bugs the same day or even before they are merged and deployed
recent QA team meeting