Do Not Fight Your Testing Tools

Gleb Bahmutov

Sr Director of Engineering, Mercari US

https://slides.com/bahmutov/no-fighting

📺 watch at https://youtu.be/b87e7k3RjOg

Heavy smoke fills the air as people cross 34th Street in Herald Square in New York City on June 6, 2023.

source: https://time.com/6285871/wildfire-smoke-photos-us-canada/

survival is possible* but we need to act now

change your life
join an organization

@bahmutov

rebellion.global www.wedonthavetime.org

www.sierraclub.org 350.org www.greenpeace.org

www.sunrisemovement.org

Why doesn't this work

gleb.dev

It used to work

gleb.dev

{testing DX}

JavaScript vs The World
JavaScript vs JavaScript vs JavaScript
Testing is broken
Testing will be alright

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

Lots Of Testing

https://slides.com/bahmutov/decks/mercari

https://cypress.tips/courses

gleb.dev

Edit and view HTML documents, 1991

source: http://digital-archaeology.org/the-nexus-browser/

Source: https://slides.com/bahmutov/http-n

gleb.dev

JavaScript!

gleb.dev

Problem: given an array of numbers, multiply each number by a constant and print the result.

var numbers = [3, 1, 7];
var constant = 2;
// 6 2 14

github.com/bahmutov/javascript-journey

procedural / imperative JS

var numbers = [3, 1, 7];
var constant = 2;
var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

Arrays: object-oriented + functional

var numbers = [3, 1, 7];
var constant = 2;
function mul(a, b) {
  return a * b;
}
function print(n) {
  console.log(n);
}
var mulBy = mul.bind(null, constant);
numbers
    .map(mulBy)
    .forEach(print);
// 6 2 14

clear "multiply then print" semantics

functional bits

Object-oriented methods

Arrays: object-oriented + functional

numbers
  .map(mulBy)
  .forEach(print);

gleb.dev

JS was influenced by Self and Scheme

numbers
  .map(mulBy)
  .forEach(print);
// same as
const a1 = numbers.map(mulBy);
a1.forEach(print)

numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)

gleb.dev

Fluent and chained

I want a value!

const X = numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)
// use x

"But what if my computation is asynchronous?"

compute((x) => {
  // x is the value
})
// runs before callback
// cannot access x

callback

gleb.dev

const X = compute()
expect(X).to.deep.equal([6, 2, 14])

What about testing?

"But what if my computation is asynchronous?"

it('computes', (done) => {
  compute((x) => {
    expect(x).to.deep.equal(...)
    done()
  })
})

gleb.dev

Promises

Single async operation with a value or an error

const p = new Promise((resolve, reject) => {
  ...
  resolve(42)
})
// after promise successfully completes
p.then(...)
 .catch(...)

var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstant, n))
        .then(print);
  };
};

Sleep, mul then print using promises

numbers.map(pauseMulAndPrint)
  .reduce(Q.when, Q())
  .done();
// ... 6 ... 2 ... 14

glebbahmutov.com/blog/chaining-promises/

it('computes', () => {
  return compute().then(x => {
    expect(x).to.deep.equal(...)
  })
})

What about testing?

gleb.dev

Async to the rescue

gleb.dev

it('computes', async () => {
  const x = await compute()
  expect(x).to.deep.equal(...)
})

Async to the rescue

gleb.dev

<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Dynamic page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Microsoft Playwright automatically retries the assertion

gleb.dev

gleb.dev

I Can't (A)wait

gleb.dev

With prices

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {

  
  await expect(async () => {
    
    
    
    
  }).toPass()
})

retry async fn

Simpler async pipelines? A dream?

Playwright (pw)

I want a value!

const X = elements
const strings = X.innerText
const prices = ...
expect(prices).to.deep.equal(...)

Promises were wrapped values

getElements()
  .then(els => els.innerText)
  .then(parsePrices)
  .then(prices => expect(prices)...)
  .catch(e => ...)

gleb.dev

In functional programming

const make = (input) =>
  Option.fromNullable(input)
    .map(parseInput)
    .flatMap((input) => Option.fromNullable(transform(input)))
    .map(print)
    .map(prettify)
    .getWithDefault("fallback");

https://swan-io.github.io/boxed/

gleb.dev

In functional reactive programming

Everything is a source of asynchronous events.

Your code is a pipeline reacting to one or more input streams and outputting another stream of events

gleb.dev

In functional reactive programming

User click events

var Rx = require('rx');

var clickEvents = Rx.Observable
  .fromEvent(document.querySelector('#btn'), 'click')

var numberEvents = Rx.Observable
  .fromArray(numbers);

function pickSecond(c, n) { return n; }

Rx.Observable.zip(clickEvents, numberEvents, 
                  pickSecond)
  .map(byConstant)
  .subscribe(print);
// prints a number on each button click

Promise

Task, Option, Result

Observable

async / await

🛑

gleb.dev

I want a value!

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

What if we accept that almost everything is async?
What if we optimize for data transformations and assertions?

gleb.dev

https://cypress.tips/courses/cypress-vs-playwright

https://slides.com/bahmutov/the-debate-cy-and-pw

gleb.dev

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Cypress (cy)

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

retry

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

retry

it('has 3 fruits', () => {
  cy.visit('public/index.html')
})

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
})

Of course the test is async

Of course loading the page is async

Of course the test must wait for the command to finish

gleb.dev

import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})

it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The Syntax

// pw

imperative syntax
promise-based

declarative syntax
reactive stream

gleb.dev https://slides.com/bahmutov/the-debate-cy-and-pw

import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})

it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The "Tinder" data flow

// pw

imperative syntax
promise-based

declarative syntax
reactive stream

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

// _ is Lodash / Ramda
_(numbers)
  .map(_.partial(mul, constant))
  .forEach(print);
// 6 2 14

// functional

for-loop vs Array.forEach

// imperative

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

https://glebbahmutov.com/blog/journey-from-procedural-to-reactive-javascript-with-stops/

const n = Number(await locator.getText())
// n is set

cy.get('#count').then(n => 
  // n is set
)

// right

Data flows left vs right

// left

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

🤯 🤬

const n = Number(await locator.getText())
// n is set

cy.get('#count').then(n => 
  // n is set
)

// right

Data flows left vs right

// left

🤯 🤬

What do you want to check on the page?

gleb.dev

cy.get('li.todo').should('have.length', 2)

There should be 2 items

gleb.dev

cy.get('li.todo')
  .should('satisfy', $li => $li.length % 2 === 0)

There should be an even number of items

gleb.dev

cy.wait('@load').its('response.body.length')
  .then(n => {
    cy.get('li.todo').should('have.length', n)
  })

There should be the same number of items as returned by the server

gleb.dev

cy.intercept('GET', '/todos', { fixture: 'three.json' })
cy.get('li.todo').should('have.length', 3)

If you can control the data in your tests, then the test syntax collapses into a simple and elegant fluent chain

$todo

gleb.dev

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

import 'cypress-map'
it('has 3 fruits', () => {
  cy.visit(index)
  cy.get('#fruits li')
    .map('innerText')
    .should('deep.equal', ['Apples', 'Grapes', 'Kiwi'])
})

functional reactive bits

if you do not have "toHaveText" assertion...

If the test looks weird or complicated...

https://docs.cypress.io/

https://cypress.tips/search

it('has 3 fruits', () => {
  try {
    cy.visit('public/index.html')
  } catch (e) {
    ...
  }
})

test('has 3 fruits', async ({ page }) => {
  try {
    await page.goto(index)
  } catch (err) {
    ...
  }
})

Side Rant: Error handling

Nope, does not work

just like it does not with

event emitters or callbacks

Ok, now what?!

gleb.dev

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Rest of your config...

  // Run your local dev server before starting the tests
  webServer: {
    command: 'npm run start',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
    timeout: 120 * 1000,
  },
});

playwright.config.ts

Tests should follow a single deterministic path

gleb.dev

I do not care how long the tests take to run.

I care how long it takes to debug a failing test

gleb.dev

Time-travel to inspect the result of each command ⏳

gleb.dev

$ npx playwright test --trace on

gleb.dev

$ npx playwright test --ui

gleb.dev

Jest + RTL + MSW vs Cypress component testing

https://dev.to/muratkeremozcan/cypress-component-testing-vs-react-test-library-the-complete-comparison-28gn

const handlers = [
  rest.get(`${process.env.REACT_APP_API_URL}/heroes`, async (_req, res, ctx) =>
    res(ctx.status(200), ctx.json(heroes))
  ),
  rest.delete(
    `${process.env.REACT_APP_API_URL}/heroes/${heroes[0].id}`, // use /.*/ for all requests
    async (_req, res, ctx) => res(ctx.status(400), ctx.json("expected error"))
  ),
];
const server = setupServer(...handlers);
beforeAll(() => {
  server.listen({
    onUnhandledRequest: "warn",
  });
});
afterEach(server.resetHandlers);
afterAll(server.close);

it("should display the hero list on render, and go through hero add & refresh flow", async () => {
  expect(await screen.findByTestId("list-header")).toBeVisible();
  expect(await screen.findByTestId("hero-list")).toBeVisible();

  await userEvent.click(await screen.findByTestId("add-button"));
  expect(window.location.pathname).toBe("/heroes/add-hero");

  await userEvent.click(await screen.findByTestId("refresh-button"));
  expect(window.location.pathname).toBe("/heroes");
});

beforeEach(() => {
  cy.intercept('GET', `${Cypress.env('API_URL')}/heroes`, {
    fixture: 'heroes.json',
  }).as('getHeroes')

  cy.wrappedMount(<Heroes />)
})

it('should display the hero list on render, and go through hero add & refresh flow', () => {
  cy.wait('@getHeroes')

  cy.getByCy('list-header').should('be.visible')
  cy.getByCy('hero-list').should('be.visible')

  cy.getByCy('add-button').click()
  cy.location('pathname').should('eq', '/heroes/add-hero')

  cy.getByCy('refresh-button').click()
  cy.location('pathname').should('eq', '/heroes')
})

"Cypress Component Testing Vs RTL and MSW: The Missing Comparison Part"

https://www.youtube.com/watch?v=rFUf7xdtt-I