Testing With Cypress vs Playwright: What Is the Difference?

Gleb Bahmutov

Sr Director of Engineering, Mercari US

https://slides.com/bahmutov/cy-vs-pw-difference

Heavy smoke fills the air as people cross 34th Street in Herald Square in New York City on June 6, 2023.

source: https://time.com/6285871/wildfire-smoke-photos-us-canada/

survival is possible* but we need to act now

change your life
join an organization

@bahmutov

rebellion.global www.wedonthavetime.org

www.sierraclub.org 350.org www.greenpeace.org

www.sunrisemovement.org

Why doesn't this work

gleb.dev

It used to work

gleb.dev

Agenda

JavaScript vs The World
JavaScript vs JavaScript vs JavaScript
PW vs Cy syntax
PW vs Cy architecture

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

Lots Of Testing

https://slides.com/bahmutov/decks/mercari

https://cypress.tips/courses

gleb.dev

Edit and view HTML documents, 1991

source: http://digital-archaeology.org/the-nexus-browser/

Source: https://slides.com/bahmutov/http-n

gleb.dev

JavaScript!

gleb.dev

Problem: given an array of numbers, multiply each number by a constant and print the result.

var numbers = [3, 1, 7];
var constant = 2;
// 6 2 14

github.com/bahmutov/javascript-journey

There is no single "correct" way to write JavaScript

procedural / imperative JS

var numbers = [3, 1, 7];
var constant = 2;
var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

Arrays: object-oriented + functional

var numbers = [3, 1, 7];
var constant = 2;
function mul(a, b) {
  return a * b;
}
function print(n) {
  console.log(n);
}
var mulBy = mul.bind(null, constant);
numbers
    .map(mulBy)
    .forEach(print);
// 6 2 14

clear "multiply then print" semantics

functional bits

Object-oriented methods

Arrays: object-oriented + functional

numbers
  .map(mulBy)
  .forEach(print);

gleb.dev

JS was influenced by Self and Scheme

numbers
  .map(mulBy)
  .forEach(print);
// same as
const a1 = numbers.map(mulBy);
a1.forEach(print)

numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)

gleb.dev

Fluent and chained

I want a value!

const X = numbers
  .map(mulBy)
  .filter(predicate)
  .map(...)
  .reduce(...)
// use x

"But what if my computation is asynchronous?"

compute((x) => {
  // x is the value
})
// runs before callback
// cannot access x

callback

gleb.dev

const X = compute()
expect(X).to.deep.equal([6, 2, 14])

What about testing?

"But what if my computation is asynchronous?"

it('computes', (done) => {
  compute((x) => {
    expect(x).to.deep.equal(...)
    done()
  })
})

gleb.dev

Promises

Single async operation with a value or an error

const p = new Promise((resolve, reject) => {
  ...
  resolve(42)
})
// after promise successfully completes
p.then(...)
 .catch(...)

var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstant, n))
        .then(print);
  };
};

Sleep, mul then print using promises

numbers.map(pauseMulAndPrint)
  .reduce(Q.when, Q())
  .done();
// ... 6 ... 2 ... 14

glebbahmutov.com/blog/chaining-promises/

it('computes', () => {
  return compute().then(x => {
    expect(x).to.deep.equal(...)
  })
})

What about testing?

gleb.dev

Async to the rescue

gleb.dev

it('computes', async () => {
  const x = await compute()
  expect(x).to.deep.equal(...)
})

Async to the rescue

gleb.dev

<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Dynamic page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

Playwright (pw)

Microsoft Playwright automatically retries the assertion

gleb.dev

gleb.dev

I Can't (A)wait

gleb.dev

With prices

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Playwright (pw)

test('has 3 fruits', async ({ page }) => {

  
  await expect(async () => {
    
    
    
    
  }).toPass()
})

retry async fn

Simpler async pipelines? A dream?

Playwright (pw)

I want a value!

const X = elements
const strings = X.innerText
const prices = ...
expect(prices).to.deep.equal(...)

Promises were wrapped values

getElements()
  .then(els => els.innerText)
  .then(parsePrices)
  .then(prices => expect(prices)...)
  .catch(e => ...)

gleb.dev

In functional programming

const make = (input) =>
  Option.fromNullable(input)
    .map(parseInput)
    .flatMap((input) => Option.fromNullable(transform(input)))
    .map(print)
    .map(prettify)
    .getWithDefault("fallback");

https://swan-io.github.io/boxed/

gleb.dev

In functional reactive programming

Everything is a source of asynchronous events.

Your code is a pipeline reacting to one or more input streams and outputting another stream of events

gleb.dev

In functional reactive programming

User click events

var Rx = require('rx');

var clickEvents = Rx.Observable
  .fromEvent(document.querySelector('#btn'), 'click')

var numberEvents = Rx.Observable
  .fromArray(numbers);

function pickSecond(c, n) { return n; }

Rx.Observable.zip(clickEvents, numberEvents, 
                  pickSecond)
  .map(byConstant)
  .subscribe(print);
// prints a number on each button click

Promise

Task, Option, Result

Observable

async / await

🛑

gleb.dev

I want a value!

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

What if we accept that almost everything is async?
What if we optimize for data transformations and assertions?

gleb.dev

https://cypress.tips/courses/cypress-vs-playwright

https://slides.com/bahmutov/the-debate-cy-and-pw

gleb.dev

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Cypress (cy)

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

retry

import 'cypress-map'

it('has 3 fruits', () => {
  cy.visit('public/index.html')
  cy.get('#fruits li')
    .map('innerText')
    .mapInvoke('split', '$')
    .mapInvoke('at', 1)
    .map(Number)
    .should('deep.equal', [1, 2, 3])
})

Elements to numbers

pipeline

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  const elements = page.getByRole('listitem')
  await expect(async () => {
    const strings = await elements.allInnerTexts()
    const prices = strings.map((s) => s.split('$')[1])
    	.map(Number)
    expect(prices).toEqual([1, 2, 3])
  }).toPass()
})

Retry

retry

it('has 3 fruits', () => {
  cy.visit('public/index.html')
})

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
})

Of course the test is async

Of course loading the page is async

Of course the test must wait for the command to finish

gleb.dev

import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})

it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The Syntax

// pw

imperative syntax
promise-based

declarative syntax
reactive stream

gleb.dev https://slides.com/bahmutov/the-debate-cy-and-pw

import { test, expect } from '@playwright/test'

test('adds todos', async ({ page, request }) => {
  await request.post('/reset', { data: { todos: [] } })
  await page.goto('/')
  // create a new todo locator
  const newTodo = page.getByPlaceholder('What needs to be done?')
  await newTodo.fill('one')
  await newTodo.press('Enter')
  await newTodo.fill('two')
  await newTodo.press('Enter')
  const todoItems = page.locator('li.todo')
  await expect(todoItems).toHaveCount(2)
})

it('adds todos', () => {
  cy.request('POST', '/reset', { todos: [] })
  cy.visit('/')
  cy.get('.new-todo').type('one{enter}').type('two{enter}')
  cy.get('li.todo').should('have.length', 2)
})

// cy

The "Tinder" data flow

// pw

imperative syntax
promise-based

declarative syntax
reactive stream

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

// _ is Lodash / Ramda
_(numbers)
  .map(_.partial(mul, constant))
  .forEach(print);
// 6 2 14

// functional

for-loop vs Array.forEach

// imperative

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

https://glebbahmutov.com/blog/journey-from-procedural-to-reactive-javascript-with-stops/

const n = Number(await locator.getText())
// n is set

cy.get('#count')
  .invoke('text')
  .then(Number)
  .then(n => {
    // n is set
  })

// right

Data flows left vs right

// left

Swipe right

(data flows to the right)

Swipe left

(data is assigned to the left)

🤯 🤬

const n = Number(await locator.getText())
// n is set

Data flows left vs right

// left

What do you want to check on the page?

gleb.dev

cy.get('#count')
  .invoke('text')
  .then(Number)
  .then(n => {
    // n is set
  })

// right

🤯 🤬

cy.get('li.todo').should('have.length', 2)

There should be 2 items

gleb.dev

cy.get('li.todo')
  .should('satisfy', $li => $li.length % 2 === 0)

There should be an even number of items

gleb.dev

cy.wait('@load').its('response.body.length')
  .then(n => {
    cy.get('li.todo').should('have.length', n)
  })

There should be the same number of items as returned by the server

gleb.dev

cy.intercept('GET', '/todos', { fixture: 'three.json' })
cy.get('li.todo').should('have.length', 3)

If you can control the data in your tests, then the test syntax collapses into a simple and elegant fluent chain

$todo

gleb.dev

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

<body>
  <h1>Fruits</h1>
  <ul>
    <li>Apples</li>
    <li>Grapes</li>
    <li>Kiwi</li>
  </ul>
</body>

Static page

test('has 3 fruits', async ({ page }) => {
  await page.goto(index)
  await expect(page.getByRole('listitem')).toHaveText([
    'Apples',
    'Grapes',
    'Kiwi',
  ])
})

import 'cypress-map'
it('has 3 fruits', () => {
  cy.visit(index)
  cy.get('#fruits li')
    .map('innerText')
    .should('deep.equal', ['Apples', 'Grapes', 'Kiwi'])
})

functional reactive bits

if you do not have "toHaveText" assertion...

If the test looks weird or complicated...

https://docs.cypress.io/

https://cypress.tips/search

Ok, I give up.

gleb.dev

https://github.com/bahmutov/cypress-await

const n = Number(await locator.getText())
// n is set

cy.get('#count')
  .invoke('text')
  .then(Number)
  .then(n => {
    // n is set
  })

// cy

// pw

🤯 🤬

npm i -D cypress-await

gleb.dev

https://github.com/bahmutov/cypress-await

const n = Number(await locator.getText())
// n is set

const n = await 
  cy.get('#count')
    .invoke('text')
    .then(Number)

// cy + cypress-awit

// pw

npm i -D cypress-await

gleb.dev

https://github.com/bahmutov/cypress-await

const n = Number(await locator.getText())
// n is set

const n = cy
    .get('#count')
    .invoke('text')
    .then(Number)

// cy + cypress-awit

// pw

npm i -D cypress-await

😍 🎁 🎉

gleb.dev

Need For Speed 🏎️

import { test } from '@playwright/test'

test('logs hello', async () => {
  console.log('hello, world')
})

it('logs hello', () => {
  console.log('hello, world')
})

// pw

// cy

// pw

// cy

Cypress iframes the app and its specs

Node.js

Playwright test code

Browser

application

Chrome Debugger Protocol

Node.js

Config / plugins file

Browser

application

Cypress test code

WebSocket

for

cy.task

Test runners architecture

Playwright

runs outside the browser
controls the browser via Chrome Debugger Protocol (FAST)

Cypress

runs inside the browser
wraps a lot of browser elements using JavaScript
creates user interface (SLOW)

Need For Speed 🏎️

I do not care how long the tests take to run.

I care how long it takes to debug a failing test

gleb.dev

Time-travel to inspect the result of each command ⏳

gleb.dev

$ npx playwright test --trace on

gleb.dev

$ npx playwright test --ui

gleb.dev

Replaying Tests: The New Hotness

https://blog.replay.io/replay-for-test-suites

gleb.dev

https://on.cypress.io/test-replay

Cypress Test Replays https://on.cypress.io/test-replay

https://blog.replay.io/replay-for-test-suites

Let me take a look. Is this why my test has failed?

gleb.dev

Final

Thoughts

gleb.dev

Different Tools Are Different

Don't Get Hanged Up On Syntax

Learn Both Cy and PW

Gleb Bahmutov

gleb.dev

👏 Thank You 👏

https://slides.com/bahmutov/cy-vs-pw-difference