How We Run A Lot Of End-to-End Tests At Mercari US

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Join others to fight the climate crisis

Gleb Bahmutov

Sr Director of Engineering

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

The same CTO

Gleb, we need our stuff to work.

  • Web app

  • ReactNative mobile app

  • APIs

  • Special projects

Agenda

  • Test speed

  • Fast(er) CI feedback

  • Test tags

  • Test reporting and triage

  • Fighting the Hydra

  • Future is 🔆

If you like it,

Put a 💍 test on it...

Gleb Beyonce Bahmutov

A typical Mercari US Cypress E2E test

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

  • Keep the tests readable
    • custom commands, utilities, plugins
  • Keep the tests readable
    • custom commands, utilities, plugins
  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
  • Keep the tests readable
    • custom commands, utilities, plugins
  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
  • Cache created data
    • users, listing
  • Keep the tests readable
    • custom commands, utilities, plugins
  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
  • Cache created data
    • users, listing​
  • Keep a test shorter than 3 minutes
    • Keep each spec shorter than 3 minutes
    • Use data-driven testing
const searches = ['Wearable', 'Running shoes', 'Dolls']

it.each(searches)(
  `Filters results by status for search: %s`,
  (searchKeyword) => {
    const url = formSearchUrl({ searchKeyword })
    cy.visitProtectedPage(url)
    ...
  })

Plus internal web application E2E tests

660 tests * 1 minute/test

11 hours to run all the tests

  1. How to run all the tests?
  2. How to run the tests for PRs?

Parallelize all the things

500+ E2E tests finish in 27 minutes using 15 CI machines

# .circleci/config.yml
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1

- cypress/run:
    name: Nightly Cypress E2E tests
    requires:
      - cypress/install
    record: true
    # split all specs across machines
    parallel: true
    # use N CircleCI machines to finish quickly
    # we can use a higher number of machines against the main deploy
    # because it has more resources compared to the preview deploys
    parallelism: 15
    tags: nightly

Use the CircleCI Cypress Orb

💻

💻

💻

💻

💻

💻

💻

💻

Dev Environment

💻

💻

💻

💻

💻

💻

💻

💻

 🔥 Dev Environment 🔥

🔥 🔥 🔥 🔥 🔥

💻

💻

💻

💻

💻

💻

💻

💻

We have to prioritize some spec files

Work locally on a single feature spec

Push code to run E2E tests on CI

Pull Request Flow

Web Repo

E2E Repo

PR

PR deploy

Trigger E2E tests

"How to Keep Cypress Tests in Another Repo While Using CircleCI"

https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci

"How to Keep Cypress Tests in Another Repo While Using GitHub Actions"

https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo/

Pull Request Flow

Web Repo

E2E Repo

PR

PR deploy

Trigger E2E tests

new / changed

specs

Pull Request Flow

Web Repo

E2E Repo

PR

PR deploy

Trigger E2E tests

💻

💻

💻

Circle CI machines

PR preview environments are isolated (GOOD), but not very powerful  even compared to the DEV environment (ughh)

new / changed

specs

Find the changed specs and run them first

# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent)
n=$(npx find-cypress-specs --branch main --parent --count)

if [ ${n} -lt 1 ]; then
  echo "No Cypress specs changed, exiting..."
  exit 0
fi

npx cypress run --record --parallel --spec ${specs}

If changed specs pass, run all or some E2E tests

Custom /cypress command in the PR comment

Test Tags

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})
describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})

Effective tags

@shipping, @sanity

@regression, @mobile

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})
$ find-cypress-specs --tags

Tag          Tests
-----------  -----
@balance     4    
@careers     2    
@helpcenter  69   
@local       19   
@login       11   
@messaging   8    
@mobile      77   
@moble       1    
@offer       6    
@payment     34   
@profile     55   
@regression  72   
@sanity      24   
@search      43   
@sell        76   
@shipping    24   
@signup      12   
@w9          10

Effective tags

@shipping, @sanity

@regression, @mobile

@profile

@sell

@login

@shipping

@payment

@sanity

@regression

all

@...

@sanity, @regression, all

features

Trigger tests from GitHub UI

Web Repo

PR

PR deploy

Trigger

tests

automatically

API1 Repo

PR

PR deploy

Trigger

tests

manually

API2 Repo

PR

PR deploy

Service X

PR

PR deploy

YOU can run the tests

@mobile is special

{
  "scripts": {
    "cy:open": "cypress open",
    "cy:open:mobile": "cypress open --config viewportWidth=400,viewportHeight=600,
      userAgent=\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) Mobile/14E304\""
  }
}

We have to run @mobile tests in a separate CI job...

# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)

if [ ${n} -lt 1 ]; then
  echo "No Cypress specs changed, exiting..."
  exit 0
fi

npx cypress run --record --parallel --spec ${specs} \
	--config userAgent="Mobile ...."

Run any changed mobile tests

# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)

if [ ${n} -lt 1 ]; then
  echo "No Cypress specs changed, exiting..."
  exit 0
fi

npx cypress run --record --parallel --spec ${specs} \
	--config userAgent="Mobile ...."

changed mobile specs

changed specs

# https://github.com/bahmutov/find-cypress-specs
specs=$(npx find-cypress-specs --branch main --parent --tagged @mobile)
n=$(npx find-cypress-specs --branch main --parent --count --tagged @mobile)

if [ ${n} -lt 1 ]; then
  echo "No Cypress specs changed, exiting..."
  exit 0
fi

npx cypress run --record --parallel --spec ${specs} \
	--config userAgent="Mobile ...."

GoogleBot is special

const setUserAgentHeader = () => {
  cy.intercept('*', (req) => {
    req.headers['user-agent'] = 'Googlebot Smartphone'
  })
}

it('renders the brand page', () => {
  setUserAgentHeader()
  // check the server-side rendered page Google sees
  cy.visitProtectedPage(brandUrl)
  cy.byAriaLabel('hamburger-menu')
  cy.byTestId('NotificationsButton').delay(5_000)
  cy.byTestId('Error500').should('not.exist')
  cy.byTestId('NameBreadcrumb', 'Apple')
})

Checking the server-side rendered pages the Google crawls

Flake Is Bad

Fighting flaky tests

  • Enable test retries, look at the Cypress Dashboard

Fighting flaky tests

  • Enable test retries, look at the Cypress Dashboard

  • Increase command timeouts where necessary

  • Command - assertion pattern

Fighting flaky tests

  • spy on GraphQL calls 🎉

"Directly Spying on GraphQL Calls Made By The Application" https://www.youtube.com/watch?v=XadOqS0YNJE

spy on GraphQL calls 🎉

spy on GraphQL calls 🎉

"Set GraphQL Operation Name As Custom Header And Use It In cy.intercept" https://www.youtube.com/watch?v=AcU5mkedchM deserves a lot more ❤️

Burn Changed / New Tests

Burn Changed / New Tests

CYPRESS_burn=5 npx cypress run ...

Test Management

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@regression', '@mobile'] },
    () => {
      ...
    }
  )   
})
// enable TestRail integration only in some workflows
// you can use CYPRESS_enableTestRail=true or use --env CLI flag
// cypress open/run --env enableTestRail=true
if (config.env.enableTestRail) {
  console.log('enableTestRail is on')
  await require('cypress-testrail-simple/src/plugin')(on, config)
}

cypress/plugins/index.js

Skip failing tests ASAP

it.skip('C1234 ...', () => {
  ...
})

TestRail

Automated E2E Tests

Manual E2E Tests

API / other tests

cheaper!

faster!

never tired!

Training

What we did:

Feature Flags

Did we just break:

  • all the tests?
  • some of the tests?
  • zero tests?

cy.byTestId('LoginSubmitButton')

cy.get('button[type=submit]')

Good!

How are features served?

  • explicit opt-in
  • percentage

Good!

Bad 🔥

# of feature flags

# of tests

  • DEV serves fixed feature flags
  • All experiments are opt-in and targeted by the user ID
  • E2E tests use each flag by explicit opt-in

Feature flags vs Tests

  • Bonus: feature flags should be retired / removed after the experiment is over

"Control LaunchDarkly From Cypress Tests"

https://glebbahmutov.com/blog/cypress-and-launchdarkly/

it('shows the feature OFF', () => {
  cy.visit('/')
  // feature A is OFF by default
})

it('shows the feature ON', () => {
  const featureFlagKey = 'featureA'
  const userId = 'USER_1234'
  // target the given user to receive the first variation of the feature flag
  cy.setFeatureFlagForUser(
    featureFlagKey,
    userId,
    variationIndex: 2, // ON
  )
  cy.visit('/')
  // feature A is ON
})
npm i -D cypress-ld-control
# configure the plugin in Cypress

"Control LaunchDarkly From Cypress Tests"

https://glebbahmutov.com/blog/cypress-and-launchdarkly/

npm i -D cypress-ld-control
# configure the plugin in Cypress
it('shows the feature OFF', () => {
  cy.visit('/')
  // feature A is OFF by default
})

it('shows the feature ON', () => {
  const featureFlagKey = 'featureA'
  const userId = 'USER_1234'
  // target the given user to receive the first variation of the feature flag
  cy.setFeatureFlagForUser(
    featureFlagKey,
    userId,
    variationIndex: 2, // ON
  )
  cy.visit('/')
  // feature A is ON
})

Automated Tests

❤️

Predictability

We now catch bugs the same day or even before they are merged and deployed

recent QA team meeting

The Future

  • Determine the specs to run based on code coverage and the changed source files
  • Cypress v10 🎉🎉🎉 component testing
  • Better triage for failed tests

How We Run A Lot Of End-to-End Tests At Mercari US