End-to-end testing is hard -

but it doesn't have to be

Gleb Bahmutov, PhD

VP of Engineering

About me:

that is where these slides are

12 people. Atlanta, Philly, Boston, LA, Chicago

Fast, easy and reliable testing for anything that runs in a browser

Cypress.io presentation at ReactiveConf 2016 https://www.youtube.com/watch?v=lK_ihqnQQEM

2016 vs 2018

October 2017: Cypress goes open source

  • MIT license
  • no feature limitations
  • high-quality docs

2016 vs 2018

2016 vs 2018

Going beyond single test run

  • Record test artifacts from any CI
  • Test parallelization, grouping, insights
  • Free for public projects

In this talk:

  • Command retries

  • Test against the browser, not code

  • Browser <-> Node

  • Declarative syntax

  • Customizing Cypress

Command retries

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

Let's fail on purpose

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

Command + assertion

command

assertion

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li', {timeout: 1000000})
    .should('have.length', 2)
})

Retry longer

helping live web application during the test

Which command is the last one?

cy.get('.todo-list')
<ul class=".todo-list">

  ...

</ul>

cy.get('.todo-list').find('li')
<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)
<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

Only the last command is retried

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

Only the last command is retried

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

cy.get('.todo-list li').should('have.length', 2)

last command

assertion

Only the last command is retried

cy.window()
  .its('app').its('$store').its('items')
  .should('have.length', 2)
cy.window()
  .its('app.$store.items')
  .should('have.length', 2)

Maybe "$store" does not exist yet, or the "app" has not yet!

if (window.Cypress) {
  window.app = app
}

app code

test code

cy.window()
  .then(...)
  .should('have.length', 2)
const myLogic = () => {
  // custom retry logic
  return new Promise(...)
}
cy.window()
  .then(myLogic)
  .should('have.length', 2)

.then() command is NOT retried

const throwDice = () => Cypress._.random(1, 6, false)
// promise-returning functions are more realistic
// in the browser world
const getDiceToBe4 = () => throwDice() === 4
  ? Cypress.Promise.resolve(4) 
  : Cypress.Promise.reject(new Error('no dice'))

Bring your own retry logic

// use promise-retry to re-execute until resolves
const promiseRetry = require('promise-retry')
const myLogic = () => promiseRetry((retry, number) => {
  return getDiceToBe4().catch(retry)
}, { factor: 1, minTimeout: 100 })

Bring your own retry logic

it('retries inside .then', function () {
  cy.then(myLogic).should('equal', 4)
})
import Convergence from '@bigtest/convergence'
it('converges inside .then', function () {
  const throwDice = () =>
    Cypress._.random(1, 6, false)
  const myLogic = () => {
    return new Convergence()
    .when(() => throwDice() === 4)
    .run()
    .then(() => 4) // have to return a value
  }
  cy.then(myLogic).should('equal', 4)
})

Bring retry logic from other testing tools 😃

Command retries

End-to-end tests must deal well with the unpredictable nature of the web.

But how does it FEEL?

When I test with Cypress

My test context is well isolated from the app's context

Gloves when not in use are always funny

DOM

Network

storage

DOM

Network

storage

framework-agnostic

implementation-agnostic

If you can write E2E tests in a framework-agnostic way

You can replace framework X with Y

(without breaking things)

What about other sides like file system or databases?

Node

Cy backend

cy.task(name, ...args)
on('task', {
  name: (...args) => ...
})

Jumping from browser to backend context in tests

result

plugins file

it('finds record in the database', () => {
  // random text to avoid confusion
  const id = Cypress._.random(1, 1e6)
  const title = `todo ${id}`
  cy.get('.new-todo').type(`${title}{enter}`)

})

Observe Database Effect

runs in the browser

drive via DOM

it('finds record in the database', () => {
  // random text to avoid confusion
  const id = Cypress._.random(1, 1e6)
  const title = `todo ${id}`
  cy.get('.new-todo').type(`${title}{enter}`)
  cy.task('hasSavedRecord', title).should('equal', true)
})

Observe Database Effect

runs in the browser

const hasRecordAsync = (title, ms) => {
  // use promise-retry or convergence
  ...
}

module.exports = (on, config) => {
  on('task', {
    hasSavedRecord (title, ms = 3000) {
      return hasRecordAsync(title, ms)
    }
  })
}

runs in Node in cypress/plugins/index.js

Observe Database Effect

task completes as soon as the server gets POST from the app and saves record to DB

task checks for wrong title and eventually times out

Node to Browser actions are hard

Node

Browser

WebDriver.execute

browser.execute(script[,argument1,...,argumentN]);

Cy.task

cy.task(name, ...args)
cy.task
cy.exec
cy.request

(and you only send data, not code)

Browser to Node is easy

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Declarative Syntax

there are no async / awaits or promise chains

Tests should read naturally

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

Puppeteer

test('My Test', async t => {
    await t
        .setNativeDialogHandler(() => true)
        .click('#populate')
        .click('#submit-button');

    const location = await t.eval(() => window.location);

    await t.expect(location.pathname)
        .eql('/testcafe/example/thank-you.html');
});

TestCafe

it('changes the URL when "awesome" is clicked', () => {
  const user = cy

  user.visit('/my/resource/path')

  user.get('.awesome-selector')
    .click()

  user.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress is like a real user

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress: all commands are in a queue

visit
get
click
it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress: all commands are in a queue

visit
get
click
url
should

Single Command Queue

visit
get
click
url
should

Like a SINGLE user driving the browser

Single Command Queue

visit
get
click
url
should

Deterministic and repeatable

Single Command Queue

visit
get
click
url
should

Lazy

Single Command Queue

visit
get
click
url
should

Lazy

Single Command Queue

visit
get
click
url
should

Lazy

Single Command Queue

visit
get
click
url
should

Lazy

Single Command Queue

visit
get
click
url
should

Lazy

Single Command Queue

visit
get
click
url
should

Commands can be retried (unlike promises)

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  const url = await cy.url()
  url.should('include', 
             '/my/resource/path#awesomeness')
})

NO!

Promises and async / await are eager single try constructs that do not work (well enough) for E2E tests

visit
get
click
url
should

Test starts

water starts flowing

can water reach finish?

visit
get
click
url
should

Test starts

events starts flowing

can events reach subscribe?

Reactive stream

range(0, 10).pipe(
  filter(x => x % 2 === 0),
  map(x => x + x),
  scan((acc, x) => acc + x, 0)
)
.subscribe(x => 
    console.log(x))
visit
get
click
url
should

writing Cypress test is like writing a single reactive stream

Cypress vs X

People testing with Selenium / WebDriver

Cypress users?

No E2E tests

Make The Test Runner Yours

before(() => {
  console.log('parent.window.document.body is', 
    parent.window.document.body)
})
before(() => {
  const $head = Cypress.$(parent.window.document.head)
  const css = '...' // new style
  $head.append(`<style 
    type="text/css" id="cypress-dark">
    ${css}
  </style>`)
})

your test code

your test code

End-to-end testing is hard -

but it doesn't have to be

Gleb Bahmutov, PhD

VP of Engineering, Cypress.io

Thank you 👏