Do Not Fight Your Testing Tools

Gleb Bahmutov

Sr Director of Engineering, Mercari US

Heavy smoke fills the air as people cross 34th Street in Herald Square in New York City on June 6, 2023.

source: https://time.com/6285871/wildfire-smoke-photos-us-canada/

survival is possible* but we need to act now

  • change your life
  • join an organization

Why doesn't this work

It used to work

{testing DX}

  • JavaScript vs The World
  • JavaScript vs JavaScript vs JavaScript
  • Testing is broken
  • Testing will be alright

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Lots Of Testing

Edit and view HTML documents, 1991

source: http://digital-archaeology.org/the-nexus-browser/

JavaScript!

Problem: given an array of numbers, multiply each number by a constant and print the result.

var numbers = [3, 1, 7];
var constant = 2;
// 6 2 14

procedural / imperative JS

var numbers = [3, 1, 7];
var constant = 2;
var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

Arrays: object-oriented + functional

var numbers = [3, 1, 7];
var constant = 2;
function mul(a, b) {
  return a * b;
}
function print(n) {
  console.log(n);
}
var mulBy = mul.bind(null, constant);
numbers
    .map(mulBy)
    .forEach(print);
// 6 2 14
  • clear "multiply then print" semantics

functional bits

Object-oriented methods

Arrays: object-oriented + functional

numbers
  .map(mulBy)
  .forEach(print);

JS was influenced by Self and Scheme

numbers
  .map(mulBy)
  .forEach(print);
// same as
const a1 = numbers.map(mulBy);
a1.forEach(print)
numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)

Fluent and chained

I want a value!

const X = numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)
// use x

"But what if my computation is asynchronous?"

compute((x) => {
  // x is the value
})
// runs before callback
// cannot access x

callback

const X = compute()
expect(X).to.deep.equal([6, 2, 14])

What about testing?

"But what if my computation is asynchronous?"

it('computes', (done) => {
  compute((x) => {
    expect(x).to.deep.equal(...)
    done()
  })
})

Promises

Single async operation with a value or an error

const p = new Promise((resolve, reject) => {
  ...
  resolve(42)
})
// after promise successfully completes
p.then(...)
 .catch(...)
var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstant, n))
        .then(print);
  };
};

Sleep, mul then print using promises

numbers.map(pauseMulAndPrint)
  .reduce(Q.when, Q())
  .done();
// ... 6 ... 2 ... 14
it('computes', () => {
  return compute().then(x => {
    expect(x).to.deep.equal(...)
  })
})

What about testing?

Async to the rescue

it('computes', async () => {
  const x = await compute()
  expect(x).to.deep.equal(...)
})

Async to the rescue

<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Dynamic page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Microsoft Playwright automatically retries the assertion

I Can't (A)wait

With prices

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {

  
  await expect(async () => {
    
    
    
    
  }).toPass()
})

retry async fn

Simpler async pipelines? A dream?

Playwright (pw)

I want a value!

const X = elements
const strings = X.innerText
const prices = ...
expect(prices).to.deep.equal(...)

Promises were wrapped values

getElements()
  .then(els => els.innerText)
  .then(parsePrices)
  .then(prices => expect(prices)...)
  .catch(e => ...)

In functional programming

const make = (input) =>
  Option.fromNullable(input)
    .map(parseInput)
    .flatMap((input) => Option.fromNullable(transform(input)))
    .map(print)
    .map(prettify)
    .getWithDefault("fallback");

In functional reactive programming

  • Everything is a source of asynchronous events.
  • Your code is a pipeline reacting to one or more input streams and outputting another stream of events

In functional reactive programming

User click events

var Rx = require('rx');

var clickEvents = Rx.Observable
  .fromEvent(document.querySelector('#btn'), 'click')

var numberEvents = Rx.Observable
  .fromArray(numbers);

function pickSecond(c, n) { return n; }

Rx.Observable.zip(clickEvents, numberEvents, 
                  pickSecond)
  .map(byConstant)
  .subscribe(print);
// prints a number on each button click

Promise 

Task, Option, Result

Observable

async / await

🛑

I want a value!

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

  • What if we accept that almost everything is async?
  • What if we optimize for data transformations and assertions?
import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Cypress (cy)

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})
import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

retry

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

Vs

retry

cy

pw

it('has 3 fruits', () => {
  cy.visit('public/index.html')
})
test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
})

Of course the test is async

Of course loading the page is async

Of course the test must wait for the command to finish

cy

pw

import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})
it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The Syntax

// pw

  • imperative syntax
  • promise-based
  • declarative syntax
  • reactive stream
import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})
it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The "Tinder" data flow

// pw

  • imperative syntax
  • promise-based
  • declarative syntax
  • reactive stream

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14
// _ is Lodash / Ramda
_(numbers)
  .map(_.partial(mul, constant))
  .forEach(print);
// 6 2 14

// functional

for-loop vs Array.forEach

// imperative

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

const n = Number(await locator.getText())
// n is set
cy.get('#count').then(n => 
  // n is set
)

// right

Data flows left vs right

// left

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

🤯 🤬

const n = Number(await locator.getText())
// n is set
cy.get('#count').then(n => 
  // n is set
)

// right

Data flows left vs right

// left

🤯 🤬

What do you want to check on the page?

cy.get('li.todo').should('have.length', 2)
There should be 2 items
cy.get('li.todo')
  .should('satisfy', $li => $li.length % 2 === 0)
There should be an even number of items
cy.wait('@load').its('response.body.length')
  .then(n => {
    cy.get('li.todo').should('have.length', n)
  })
There should be the same number of items as returned by the server
cy.intercept('GET', '/todos', { fixture: 'three.json' })
cy.get('li.todo').should('have.length', 3)

If you can control the data in your tests, then the test syntax collapses into a simple and elegant fluent chain

$todo

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})
<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})
import 'cypress-map'
it('has 3 fruits', () => {
  cy.visit(index)
  cy.get('#fruits li')
    .map('innerText')
    .should('deep.equal', ['Apples', 'Grapes', 'Kiwi'])
})

functional reactive bits

if you do not have "toHaveText" assertion...

If the test looks weird or complicated...

it('has 3 fruits', () => {
  try {
    cy.visit('public/index.html')
  } catch (e) {
    ...
  }
})
test('has 3 fruits', async ({ page }) => {
  try {
    await page.goto(index)
  } catch (err) {
    ...
  }
})

Side Rant: Error handling

Nope, does not work

just like it does not with

event emitters or callbacks

Ok, now what?!

cy

pw

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Rest of your config...

  // Run your local dev server before starting the tests
  webServer: {
    command: 'npm run start',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
    timeout: 120 * 1000,
  },
});

playwright.config.ts

Tests should follow a single deterministic path

I do not care how long the tests take to run.

I care how long it takes to debug a failing test

Time-travel to inspect the result of each command ⏳

$ npx playwright test --trace on
$ npx playwright test --ui

Jest + RTL + MSW vs Cypress component testing

const handlers = [
  rest.get(`${process.env.REACT_APP_API_URL}/heroes`, async (_req, res, ctx) =>
    res(ctx.status(200), ctx.json(heroes))
  ),
  rest.delete(
    `${process.env.REACT_APP_API_URL}/heroes/${heroes[0].id}`, // use /.*/ for all requests
    async (_req, res, ctx) => res(ctx.status(400), ctx.json("expected error"))
  ),
];
const server = setupServer(...handlers);
beforeAll(() => {
  server.listen({
    onUnhandledRequest: "warn",
  });
});
afterEach(server.resetHandlers);
afterAll(server.close);

it("should display the hero list on render, and go through hero add & refresh flow", async () => {
  expect(await screen.findByTestId("list-header")).toBeVisible();
  expect(await screen.findByTestId("hero-list")).toBeVisible();

  await userEvent.click(await screen.findByTestId("add-button"));
  expect(window.location.pathname).toBe("/heroes/add-hero");

  await userEvent.click(await screen.findByTestId("refresh-button"));
  expect(window.location.pathname).toBe("/heroes");
});
beforeEach(() => {
  cy.intercept('GET', `${Cypress.env('API_URL')}/heroes`, {
    fixture: 'heroes.json',
  }).as('getHeroes')

  cy.wrappedMount(<Heroes />)
})

it('should display the hero list on render, and go through hero add & refresh flow', () => {
  cy.wait('@getHeroes')

  cy.getByCy('list-header').should('be.visible')
  cy.getByCy('hero-list').should('be.visible')

  cy.getByCy('add-button').click()
  cy.location('pathname').should('eq', '/heroes/add-hero')

  cy.getByCy('refresh-button').click()
  cy.location('pathname').should('eq', '/heroes')
})

"Cypress Component Testing Vs RTL and MSW: The Missing Comparison Part"

Jest

dumps

the current

HTML

"Cypress Component Testing Vs RTL and MSW: The Missing Comparison Part"

Cypress

shows what

each test

command

did or did not do

Time-Travel From The Future

Let me take a look. Is this why my test has failed?

Final

Thoughts

Past Predictions

  • Component testing in Cypress will be big
  • Capturing everything during tests will be big

Gleb at AssertJS 2018

https://slides.com/bahmutov/assertjs

  • Playwright needs UI mode
  • Cypress needs trace capture

Gleb at Global Test Summit 2022

https://slides.com/bahmutov/the-debate-cy-and-pw

Past Predictions

Predictions

  • Diffing captured information >>> capture all information
  • The line between monitoring and testing tools will become blurry

Gleb at Refactor DX 2023

https://slides.com/bahmutov/no-fighting

*

* past performance does not guarantee future results. For entertainment purposes only

Do Not Fight Your Testing Tools

Gleb Bahmutov

gleb.dev

👏 Thank You 👏