Tips &amp; Tricks For Writing Fast And Maintainable Front-End Tests

www.youtube.com/glebbahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

cypress.tips

https://cypress.tips/courses

Gleb Bahmutov

Sr Director of Engineering

A typical Mercari US Cypress E2E test

Plus internal web application E2E tests

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

Web app tests should run in the browser

sure, but ...

import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

...

it("changes value when clicked", () => {
  ...

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  ...
  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });
});

You probably are not testing what you think you are testing...

Node code

Should be tested in Node

Browser code

Should be tested in a browser

Let's see some front-end tests

Let's test

the game

modes

npm start

npx cypress open

npm i -D cypress

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
  },

  component: {
    ...
  },
})

cypress.config.js

it('changes the number of filled cells', () => {
  cy.visit('/')
  cy.get('select[name=status__difficulty-select]').should(
    'have.value',
    'Easy',
  )
  cy.get('.game__cell--filled').should('have.length', 45)

  cy.get('select[name=status__difficulty-select]').select(
    'Medium',
  )
  cy.get('.game__cell--filled').should('have.length', 40)

  cy.get('select[name=status__difficulty-select]').select(
    'Hard',
  )
  cy.get('.game__cell--filled').should('have.length', 30)
})

cypress/e2e/modes.cy.js

The game modes test

See

the game

like a player user

Can we

test

the game

play?

It is hard

to control

the random

game board

😢

We can

use the

Hint

button

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

Click on the cell

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

Click on the hint

it('fills each empty cell using Hint', () => {
  cy.visit('/')
  cy.get('.game__cell.game__cell--filled').should(
    'have.length',
    45,
  )
  cy.get('.game__cell')
    .not('.game__cell--filled')
    .each(($cell) => {
      cy.wrap($cell, { log: false }).click()
      cy.get('.status__action-hint').click()
    })

  cy.contains('.overlay__text', 'You solved it').should(
    'be.visible',
  )
})

cypress/e2e/hint.cy.js

The hint test

Random data
External data
Timers and clocks
Conditional testing
Unclear test requirements

Making E2E hard

Let's

test the

Timer

it('Timer shows 10 seconds', () => {
  cy.visit('/')
  for (let k = 0; k < 10; k++) {
    cy.contains('.status__time', `00:0${k}`)
  }
})

cypress/e2e/timer-clock.cy.js

How does the Timer component show 15 minutes?

it('Timer shows 15 minutes', () => {
  cy.visit('/')
  cy.contains('.status__time', '15:00', {
    timeout: 900_000,
  })
})

Wait 15 minutes...

There Must Be A Better Way

How do we test smaller pieces of code?

import React, { useState, useEffect } from 'react'
import { useSudokuContext } from '../context/SudokuContext'
import moment from 'moment'

export const formatTime = ({ hours, minutes, seconds }) => {
  if (typeof seconds === 'undefined') {
    return '00:00'
  }
  let stringTimer = ''

  stringTimer += hours ? '' + hours + ':' : ''
  stringTimer += minutes ? (minutes < 10 ? '0' : '') + minutes + ':' : '00:'
  stringTimer += seconds < 10 ? '0' + seconds : seconds

  return stringTimer
}

export const Timer = (props) => { ... }

src/components/Timer.js

import { formatTime } from '../../src/components/Timer'

it('formats the time', () => {
  expect(formatTime({})).to.equal('00:00')
  expect(formatTime({ seconds: 3 })).to.equal('00:03')
  expect(formatTime({ minutes: 55, seconds: 3 })).to.equal(
    '55:03',
  )
  expect(
    formatTime({ hours: 110, minutes: 55, seconds: 3 }),
  ).to.equal('110:55:03')
})

cypress/e2e/format-time.cy.js

import { formatTime } from '../../src/components/Timer'

it('formats the time', () => {
  expect(formatTime({})).to.equal('00:00')
  expect(formatTime({ seconds: 3 })).to.equal('00:03')
  expect(formatTime({ minutes: 55, seconds: 3 })).to.equal(
    '55:03',
  )
  expect(
    formatTime({ hours: 110, minutes: 55, seconds: 3 }),
  ).to.equal('110:55:03')
})

cypress/e2e/format-time.cy.js

import { Timer } from '../../src/components/Timer'

it('shows the time', () => {
  <Timer minutes={15} />
  cy.contains('.status__time', '15:00')
})

I want this!

Component Test

// imports framework-specific component
// needs to bundle all its dependencies
// needs to "start" the framework
import { Timer } from '../../src/components/Timer'

Component Test

We are not in cy.visit('Kansas') anymore

~~npm start~~

npx cypress open

https://on.cypress.io/component-testing

Configure Cypress component testing

Supported frameworks

https://on.cypress.io/component-testing

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
  },
  component: {
    devServer: {
      framework: 'create-react-app',
      bundler: 'webpack',
    },
  },
})

cypress.config.js

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

58ms test
live app
DOM snapshots

See the component running

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Cypress "understands" how to bundle and mount your framework component

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Import app or component CSS and use the markup close to the app

without App.css or markup

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Import app or component CSS and use the markup close to the app

with App.css and markup

import React from 'react'
import { Timer } from './Timer'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'
import moment from 'moment'

describe('Timer', () => {
  it('sets the clock to the given value', () => {
    const timeGameStarted = moment().subtract(
      900,
      'seconds',
    )

    cy.mount(
      <SudokuContext.Provider value={{ timeGameStarted }}>
        <section className="status">
          <Timer />
        </section>
      </SudokuContext.Provider>,
    )
    cy.contains('15:00')
  })
})

src/components/Timer.cy.js

Pass the component data using props or other mechanisms

Timer

Numbers

Difficulty

StatusSection

GameSection

Game

App

Overlay

formatTime

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

src/components/Numbers.cy.js

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

The rest of the test is Cypress API only

src/components/Numbers.cy.js

import React from 'react'
import { Numbers } from './Numbers'
import '../App.css'
import { SudokuContext } from '../context/SudokuContext'

it('shows the selected number', () => {
  cy.mount(
    <SudokuContext.Provider
      value={{ numberSelected: '8' }}
    >
      <div className="innercontainer">
        <section className="status">
          <Numbers
            onClickNumber={cy.stub().as('click')}
          />
        </section>
      </div>
    </SudokuContext.Provider>,
  )

  cy.contains('.status__number', '8').should(
    'have.class',
    'status__number--selected',
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should(
    'have.been.calledOnceWithExactly',
    '9',
  )
})

Props + Provider are framework specific

The rest of the test is Cypress API only

src/components/Numbers.cy.js

Don't make me remember framework-specific test syntax...

import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

...

it("changes value when clicked", () => {
  ...

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  ...
  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });
});

I just want to click...

https://glebbahmutov.com/blog/cypress-component-testing-vs-angular-test-harness/

React test utils

import React from "react";
import { render, unmountComponentAtNode } from "react-dom";
import { act } from "react-dom/test-utils";

import Toggle from "./toggle";

let container = null;
beforeEach(() => {
  // setup a DOM element as a render target
  container = document.createElement("div");
  document.body.appendChild(container);
});

afterEach(() => {
  // cleanup on exiting
  unmountComponentAtNode(container);
  container.remove();
  container = null;
});

it("changes value when clicked", () => {
  const onChange = jest.fn();
  act(() => {
    render(<Toggle onChange={onChange} />, container);
  });

  // get a hold of the button element, and trigger some clicks on it
  const button = document.querySelector("[data-testid=toggle]");
  expect(button.innerHTML).toBe("Turn on");

  act(() => {
    button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
  });

  expect(onChange).toHaveBeenCalledTimes(1);
  expect(button.innerHTML).toBe("Turn off");

  act(() => {
    for (let i = 0; i < 5; i++) {
      button.dispatchEvent(new MouseEvent("click", { bubbles: true }));
    }
  });

  expect(onChange).toHaveBeenCalledTimes(6);
  expect(button.innerHTML).toBe("Turn on");
});

https://reactjs.org/docs/testing-recipes.html#events

import React from "react";
import Toggle from "./toggle";

it("changes value when clicked", () => {
  cy.mount(<Toggle onChange={cy.stub().as('change')} />);

  // get a hold of the button element, and trigger some clicks on it
  cy.contains("[data-testid=toggle]", "Turn on").click()
  cy.get('@change').should('have.been.calledOnce')
  cy.contains("[data-testid=toggle]", "Turn off")
    .click()
    .click()
    .click()
    .click()
    .click()

  cy.get('@change').its('callCount').should('eq', 6)
  cy.contains("[data-testid=toggle]", "Turn on")
});

equivalent Cypress component test

https://on.cypress.io/api

expect(formatTime({ seconds: 3 }))
  .to.equal('00:03')

Unit test

import { Component } from './Component'
cy.mount(<Component props=... />)
cy.get(...).click()

Component test

cy.visit('/')
cy.get(...).click()

End-to-End test

Small chunks of code like functions and classes

Front-end React / Angular / Vue / X components
Easy to test edge conditions

Web application
Easy to test the entire user flow

Unit tests

Component tests

End-to-end tests

346

WIP

695

At Mercari US

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc

cy.request({
  method: 'POST',
  url: '/user',
  body: {
    ...
  },
  auth: ...
})

🧑‍🚒 The original way to bypass the human UI

Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc
Cache created data
- users, listing

cy.dataSession({
  name: 'user',
  setup () {
    ...
  },
  validate () {
    ...
  },
  recreate () {
    ...
  }
})

https://github.com/bahmutov/cypress-data-session

Do as much as possible via API calls
- login, add an address, add a credit card, create a listing, etc
Cache created data
- users, listing
Keep a test shorter than 3 minutes
- Keep each spec shorter than 3 minutes

https://on.cypress.io/parallelization

700 tests * 1 minute/test

≅

12 hours to run all the tests

How to run all the tests?
How to run the tests for PRs?

Parallelize all the things

500+ E2E tests finish in 27 minutes using 15 CI machines

Test

data

cleanup

job

# .circleci/config.yml
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1

- cypress/run:
    name: Nightly Cypress E2E tests
    requires:
      - cypress/install
    record: true
    # split all specs across machines
    parallel: true
    # use N CircleCI machines to finish quickly
    # we can use a higher number of machines against the main deploy
    # because it has more resources compared to the preview deploys
    parallelism: 15
    tags: nightly

Use the CircleCI Cypress Orb

Work locally on a single feature spec

Push code to run E2E tests on CI

Installation

Writing tests

Running tests

Debugging tests

Maintenance

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

CI Setup

What Is Fast?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

CI Setup

The test runner

What Is Fast?

Tips & Tricks For Writing Fast And Maintainable Front-End Tests

by someone else

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

Keep the tests readable
- custom commands, utilities, plugins

Sprinkle page abstractions when needed.

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

A typical Cypress test

Custom commands

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

A typical Cypress test

No page abstraction

Make The Errors Actionable

https://www.cypress.io/blog/2017/07/26/good-error-messages/

Why are we still on this page?

Maybe one of these API calls failed?

    cy.interceptGraphQL('editDraftItemMutation')
    cy.byTestId('NewListerSidebarSell').click()
    cy.waitGraphQL('@editDraftItemMutation')

Confirm API call worked

We should investigate this API

https://www.cypress.io/blog/2017/07/26/good-error-messages/

Notify People Who Can Fix It

https://github.com/cypress-io/cypress-grep

Learn more:

https://glebbahmutov.com/blog/tag-tests/

https://github.com/bahmutov/find-cypress-specs

describe('Shipping', { tags: '@shipping' }, () => {
  it(
    'C1234 uses the default Mercari shipping',
    { tags: ['@sanity', '@mobile'] },
    () => {
      ...
    }
  )   
})

Effective tags

@shipping, @sanity, @mobile

{
  "@shipping": "#slack-e2e-channel @gleb @john",
  ...
}

https://github.com/bahmutov/cypress-slack-notify

Notify the right team when an E2E fails 🚨

Stub Stable APIs

Timer

Numbers

Difficulty

StatusSection

GameSection

Game

App

Overlay

formatTime

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

https://on.cypress.io/intercept

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

import { Game } from './Game'
import { SudokuProvider } from './context/SudokuContext'
import { WinProvider } from './context/WinContext'
import { starting, solved } from '../cypress/fixtures/sudoku.json'

it('plays the game', () => {
  cy.mount(
    <SudokuProvider>
      <WinProvider>
        <Game initArray={starting} solvedArray={solved} />
      </WinProvider>
    </SudokuProvider>,
  )
  cy.get('.game__cell:not(.game__cell--filled)').should(
    'have.length',
    3,
  )
  starting.forEach((cell, index) => {
    if (cell === '0') {
      cy.get('.game__cell').eq(index).click()
      cy.contains('.status__number', solved[index])
        .click()
        .wait(500, { log: false })
    }
  })

  cy.contains('.overlay__text', 'You solved it').should('be.visible')
})

src/Game.cy.js

export const Overlay = (props) => {
  const { loading, data } = useFetch(
    props.overlay && props.time ? '/times/' + props.time : null,
  )
  const className = props.overlay
    ? 'overlay overlay--visible'
    : 'overlay'
  return (
    <div className={className} onClick={props.onClickOverlay}>
      <h2 className="overlay__text">
        <div className="overlay__greeting">
          You <span className="overlay__textspan1">solved</span>{' '}
          <span className="overlay__textspan2">it!</span>
        </div>
        {loading && (
          <div className="overlay__loading">Loading...</div>
        )}

        {data.length > 0 && (
          <ul className="overlay__times">
            {data.map((item, index) => {
              return (
                <li
                  key={index}
                  className={item.current ? 'overlay__current' : ''}
                >
                  {formatTime(item)}
                </li>
              )
            })}
          </ul>
        )}
      </h2>
    </div>
  )
}

src/components/Overlay.js

Stub the window.fetch?

Stub an immediate import?

Stub a method in some internal class?

it('shows the top times', () => {
  cy.intercept('GET', '/times/90', {
    fixture: 'times.json',
  }).as('scores')
  cy.mount(<Overlay overlay={true} time={90} />)
  cy.wait('@scores')
  cy.get('.overlay__times li').should('have.length', 4)
  cy.contains('.overlay__times li', '01:30').should(
    'have.class',
    'overlay__current',
  )
})

https://cypress.tips/courses/network-testing

Write and run tests in the browser
- End-to-end, component, and unit tests
There are different kinds of speed
- Writing the tests
- Running the tests
- Debugging the failures
Maintenance is hard
- Optimize for understanding

https://slides.com/bahmutov/decks/mercari

👏 Thank you! 👏

https://slides.com/bahmutov/tips-tricks