Tips & Tricks For Writing Fast And Maintainable Front-End Tests

Gleb Bahmutov

@bahmutov

VOTE

@bahmutov

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

Web app tests should run in the browser

sure, but ...

import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

...

it("changes value when clicked", () => {
  ...

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  ...
  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });
});

You probably are not testing what you think you are testing...

Node code

Should be tested in Node

Browser code

Should be tested in a browser

Let's see some front-end tests

Let's test

the game

modes

npm start

npx cypress open

npm i -D cypress

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
  },

  component: {
    ...
  },
})

cypress.config.js

it('changes the number of filled cells', () => {
  cy.visit('/')
  cy.get('select[name=status__difficulty-select]').should(
    'have.value',
    'Easy',
  )
  cy.get('.game__cell--filled').should('have.length', 45)

  cy.get('select[name=status__difficulty-select]').select(
    'Medium',
  )
  cy.get('.game__cell--filled').should('have.length', 40)

  cy.get('select[name=status__difficulty-select]').select(
    'Hard',
  )
  cy.get('.game__cell--filled').should('have.length', 30)
})

cypress/e2e/modes.cy.js

The game modes test

See

the game

like a player user

Can we

test

the game

play?

It is hard

to control

the random

game board

😢

We can

use the

Hint

button

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

Click on the cell

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

Click on the hint

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

The hint test

  • Random data

  • External data

  • Timers and clocks

  • Conditional testing

  • Unclear test requirements

Making E2E hard

Let's

test the

Timer

it('Timer shows 10 seconds', () => {
  cy.visit('/')
  for (let k = 0; k < 10; k++) {
    cy.contains('.status__time', `00:0${k}`)
  }
})

cypress/e2e/timer-clock.cy.js

How does the Timer component show 15 minutes?

it('Timer shows 15 minutes', () => {
  cy.visit('/')
  cy.contains('.status__time', '15:00', {
    timeout: 900_000,
  })
})

Wait 15 minutes...

There Must Be A Better Way

How do we test smaller pieces of code?

import React, { useState, useEffect } from 'react'
import { useSudokuContext } from '../context/SudokuContext'
import moment from 'moment'

export const formatTime = ({ hours, minutes, seconds }) => {
  if (typeof seconds === 'undefined') {
    return '00:00'
  }
  let stringTimer = ''

  stringTimer += hours ? '' + hours + ':' : ''
  stringTimer += minutes ? (minutes < 10 ? '0' : '') + minutes + ':' : '00:'
  stringTimer += seconds < 10 ? '0' + seconds : seconds

  return stringTimer
}

export const Timer = (props) => { ... }

src/components/Timer.js

import { formatTime } from '../../src/components/Timer'

it('formats the time', () => {
  expect(formatTime({})).to.equal('00:00')
  expect(formatTime({ seconds: 3 })).to.equal('00:03')
  expect(formatTime({ minutes: 55, seconds: 3 })).to.equal(
    '55:03',
  )
  expect(
    formatTime({ hours: 110, minutes: 55, seconds: 3 }),
  ).to.equal('110:55:03')
})

cypress/e2e/format-time.cy.js

import { formatTime } from '../../src/components/Timer'

it('formats the time', () => {
  expect(formatTime({})).to.equal('00:00')
  expect(formatTime({ seconds: 3 })).to.equal('00:03')
  expect(formatTime({ minutes: 55, seconds: 3 })).to.equal(
    '55:03',
  )
  expect(
    formatTime({ hours: 110, minutes: 55, seconds: 3 }),
  ).to.equal('110:55:03')
})

cypress/e2e/format-time.cy.js

import { Timer } from '../../src/components/Timer'

it('shows the time', () => {
  <Timer minutes={15} />
  cy.contains('.status__time', '15:00')
})

I want this!

Component Test

// imports framework-specific component
// needs to bundle all its dependencies
// needs to "start" the framework
import { Timer } from '../../src/components/Timer'

Component Test

We are not in cy.visit('Kansas') anymore

npm start

npx cypress open

Configure Cypress component testing

Supported frameworks

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
  },
  component: {
    devServer: {
      framework: 'create-react-app',
      bundler: 'webpack',
    },
  },
})

cypress.config.js

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

  • 58ms test
  • live app
  • DOM snapshots

See the component running

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Cypress "understands"  how to bundle and mount your framework component

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Import app or component CSS and use the markup close to the app

without App.css or markup

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Import app or component CSS and use the markup close to the app

with App.css and markup

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Pass the component data using props or other mechanisms

Timer

Numbers

Difficulty

StatusSection

GameSection

Game

App

Overlay

formatTime

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

src/components/Numbers.cy.js

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

The rest of the test is Cypress API only

src/components/Numbers.cy.js

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

The rest of the test is Cypress API only

src/components/Numbers.cy.js

Don't make me remember framework-specific test syntax...

import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

...

it("changes value when clicked", () => {
  ...

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  ...
  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });
});

I just want to click...

React test utils

import React from "react";
import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

import Toggle from "./toggle";

let container = null;
beforeEach(() => {
  // setup a DOM element as a render target
  container = document.createElement("div");
  document.body.appendChild(container);
});

afterEach(() => {
  // cleanup on exiting
  unmountComponentAtNode(container);
  container.remove();
  container = null;
});

it("changes value when clicked", () => {
  const onChange = jest.fn();
  act(() => {
    render(<Toggle onChange={onChange} />, container);
  });

  // get a hold of the button element, and trigger some clicks on it
  const button = document.querySelector("[data-testid=toggle]");
  expect(button.innerHTML).toBe("Turn on");

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  expect(onChange).toHaveBeenCalledTimes(1);
  expect(button.innerHTML).toBe("Turn off");

  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });

  expect(onChange).toHaveBeenCalledTimes(6);
  expect(button.innerHTML).toBe("Turn on");
});
import React from "react";
import Toggle from "./toggle";

it("changes value when clicked", () => {
  cy.mount(<Toggle onChange={cy.stub().as('change')} />);

  // get a hold of the button element, and trigger some clicks on it
  cy.contains("[data-testid=toggle]", "Turn on").click()
  cy.get('@change').should('have.been.calledOnce')
  cy.contains("[data-testid=toggle]", "Turn off")
    .click()
    .click()
    .click()
    .click()
    .click()

  cy.get('@change').its('callCount').should('eq', 6)
  cy.contains("[data-testid=toggle]", "Turn on")
});

equivalent Cypress component test

https://on.cypress.io/api

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

  • Small chunks of code like functions and classes
  • Front-end React / Angular / Vue / X components
  • Easy to test edge conditions
  • Web application
  • Easy to test the entire user flow

Unit tests

Component tests

End-to-end tests

346

WIP

695

At Mercari US

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
cy.request({
  method: 'POST',
  url: '/user',
  body: {
    ...
  },
  auth: ...
})

🧑‍🚒 The original way to bypass the human UI

  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
  • Cache created data
    • users, listing​
cy.dataSession({
  name: 'user',
  setup () {
    ...
  },
  validate () {
    ...
  },
  recreate () {
    ...
  }
})
  • Do as much as possible via API calls
    • login, add an address, add a credit card, create a listing, etc
  • Cache created data
    • users, listing​
  • Keep a test shorter than 3 minutes
    • Keep each spec shorter than 3 minutes

700 tests * 1 minute/test

12 hours to run all the tests

  1. How to run all the tests?
  2. How to run the tests for PRs?

Parallelize all the things

500+ E2E tests finish in 27 minutes using 15 CI machines

Test 

data

cleanup

job

# .circleci/config.yml
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1

- cypress/run:
    name: Nightly Cypress E2E tests
    requires:
      - cypress/install
    record: true
    # split all specs across machines
    parallel: true
    # use N CircleCI machines to finish quickly
    # we can use a higher number of machines against the main deploy
    # because it has more resources compared to the preview deploys
    parallelism: 15
    tags: nightly

Use the CircleCI Cypress Orb

Work locally on a single feature spec

Push code to run E2E tests on CI

Installation

Writing tests

Running tests

Debugging tests

Maintenance

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

The test runner

What Is Fast?

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

by someone else

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

  • Keep the tests readable
    • custom commands, utilities, plugins

Sprinkle page abstractions when needed.

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

A typical Cypress test

Custom commands

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

A typical Cypress test

No page abstraction

Make The Errors Actionable

Why are we still on this page?

Why are we still on this page?

Maybe one of these API calls failed?

    cy.interceptGraphQL('editDraftItemMutation')
    cy.byTestId('NewListerSidebarSell').click()
    cy.waitGraphQL('@editDraftItemMutation')

Confirm API call worked

We should investigate this API

Notify People Who Can Fix It

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@mobile'] },
    () => {
      ...
    }
  )   
})

Effective tags

@shipping, @sanity, @mobile

{
  "@shipping": "#slack-e2e-channel @gleb @john",
  ...
}

Notify the right team when an E2E fails 🚨

Stub Stable APIs

Timer

Numbers

Difficulty

StatusSection

GameSection

Game

App

Overlay

formatTime

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

src/Game.cy.js

export const Overlay = (props) => {
  const { loading, data } = useFetch(
    props.overlay && props.time ? '/times/' + props.time : null,
  )
  const className = props.overlay
    ? 'overlay overlay--visible'
    : 'overlay'
  return (
    <div className={className} onClick={props.onClickOverlay}>
      <h2 className="overlay__text">
        <div className="overlay__greeting">
          You <span className="overlay__textspan1">solved</span>{' '}
          <span className="overlay__textspan2">it!</span>
        </div>
        {loading && (
          <div className="overlay__loading">Loading...</div>
        )}

        {data.length > 0 && (
          <ul className="overlay__times">
            {data.map((item, index) => {
              return (
                <li
                  key={index}
                  className={item.current ? 'overlay__current' : ''}
                >
                  {formatTime(item)}
                </li>
              )
            })}
          </ul>
        )}
      </h2>
    </div>
  )
}

src/components/Overlay.js

Stub the window.fetch?

Stub an immediate import?

Stub a method in some internal class?

it('shows the top times', () => {
  cy.intercept('GET', '/times/90', {
    fixture: 'times.json',
  }).as('scores')
  cy.mount(<Overlay overlay={true} time={90} />)
  cy.wait('@scores')
  cy.get('.overlay__times li').should('have.length', 4)
  cy.contains('.overlay__times li', '01:30').should(
    'have.class',
    'overlay__current',
  )
})
  • Write and run tests in the browser
    • End-to-end, component, and unit tests
  • There are different kinds of speed
    • Writing the tests
    • Running the tests
    • Debugging the failures
  • Maintenance is hard
    • Optimize for understanding

👏 Thank you! 👏

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

By Gleb Bahmutov

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

In this talk, I will share my tips and tricks for writing frontend tests that clearly communicate the software's intent. We will look at both the end-to-end and component tests and the fuzzy boundary between those two types of tests. We will look at tricks to ensure the tests do not spuriously flake when running on the CI system. Finally, I will show my trick for making the tests useful to more people outside the QA team; if you believe like I do that the tests are a working description of the current state of the web application, then any discussion and change to that system must take the tests into the account. Watch at https://youtu.be/3Es17_P7DV8

  • 938