End-to-end testing is hard -

but it doesn't have to be

Gleb Bahmutov, PhD

VP of Engineering

video at: https://www.youtube.com/watch?v=swpz0H0u13k

@bahmutov

glebbahmutov.com

github.com/bahmutov

slides.com/bahmutov

About me:

that is where these slides are

12 people. Atlanta, Philly, Boston, LA, Chicago

Fast, easy and reliable testing for anything that runs in a browser

Cypress.io presentation at ReactiveConf 2016 https://www.youtube.com/watch?v=lK_ihqnQQEM

2016 vs 2018

October 2017: Cypress goes open source

https://www.cypress.io/blog/2017/10/10/cypress-is-now-public-beta/

  • MIT license
  • no feature limitations
  • high-quality docs

2016 vs 2018

http://www.timqian.com/star-history/#cypress-io/cypress

2016 vs 2018

Going beyond single test run

  • Record test artifacts from any CI
  • Test parallelization, grouping, insights
  • Free for public projects

In this talk:

  • Command retries

  • Test against the browser, not code

  • Browser <-> Node

  • Declarative syntax

  • Customizing Cypress

Command retries

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

Let's fail on purpose

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

Command + assertion

command

assertion

it('adds items', function () {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')
    .type('todo C{enter}')
    .type('todo D{enter}')
  cy.get('.todo-list li', {timeout: 1000000})
    .should('have.length', 2)
})

Retry longer

helping live web application during the test

https://on.cypress.io/configuration#Timeouts

Which command is the last one?

cy.get('.todo-list')

<ul class=".todo-list">

  ...

</ul>

cy.get('.todo-list').find('li')

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

Only the last command is retried

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

<ul class=".todo-list">

  <li>...</li>

  <li>...</li>

  ...

</ul>

Only the last command is retried

cy.get('.todo-list').find('li').should('have.length', 2)

last command

assertion

cy.get('.todo-list li').should('have.length', 2)

last command

assertion

Only the last command is retried

cy.window()
  .its('app').its('$store').its('items')
  .should('have.length', 2)

cy.window()
  .its('app.$store.items')
  .should('have.length', 2)

Maybe "$store" does not exist yet, or the "app" has not yet!

if (window.Cypress) {
  window.app = app
}

app code

test code

cy.window()
  .then(...)
  .should('have.length', 2)

const myLogic = () => {
  // custom retry logic
  return new Promise(...)
}
cy.window()
  .then(myLogic)
  .should('have.length', 2)

.then() command is NOT retried

const throwDice = () => Cypress._.random(1, 6, false)
// promise-returning functions are more realistic
// in the browser world
const getDiceToBe4 = () => throwDice() === 4
  ? Cypress.Promise.resolve(4) 
  : Cypress.Promise.reject(new Error('no dice'))

Bring your own retry logic

// use promise-retry to re-execute until resolves
const promiseRetry = require('promise-retry')
const myLogic = () => promiseRetry((retry, number) => {
  return getDiceToBe4().catch(retry)
}, { factor: 1, minTimeout: 100 })

Bring your own retry logic

it('retries inside .then', function () {
  cy.then(myLogic).should('equal', 4)
})
import Convergence from '@bigtest/convergence'
it('converges inside .then', function () {
  const throwDice = () =>
    Cypress._.random(1, 6, false)
  const myLogic = () => {
    return new Convergence()
    .when(() => throwDice() === 4)
    .run()
    .then(() => 4) // have to return a value
  }
  cy.then(myLogic).should('equal', 4)
})

Bring retry logic from other testing tools 😃

Command retries

End-to-end tests must deal well with the unpredictable nature of the web.

But how does it FEEL?

@bahmutov

When I test with Cypress

@bahmutov

My test context is well isolated from the app's context

Gloves when not in use are always funny

@bahmutov

DOM

Network

storage

DOM

Network

storage

framework-agnostic

implementation-agnostic

@bahmutov

If you can write E2E tests in a framework-agnostic way

You can replace framework X with Y

@bahmutov

(without breaking things)

What about other sides like file system or databases?

@bahmutov

Node

Cy backend

cy.task(name, ...args)
on('task', {
  name: (...args) => ...
})

Jumping from browser to backend context in tests

@bahmutov

result

plugins file

it('finds record in the database', () => {
  // random text to avoid confusion
  const id = Cypress._.random(1, 1e6)
  const title = `todo ${id}`
  cy.get('.new-todo').type(`${title}{enter}`)

})

Observe Database Effect

runs in the browser

drive via DOM

@bahmutov

it('finds record in the database', () => {
  // random text to avoid confusion
  const id = Cypress._.random(1, 1e6)
  const title = `todo ${id}`
  cy.get('.new-todo').type(`${title}{enter}`)
  cy.task('hasSavedRecord', title).should('equal', true)
})

Observe Database Effect

runs in the browser

@bahmutov

const hasRecordAsync = (title, ms) => {
  // use promise-retry or convergence
  ...
}

module.exports = (on, config) => {
  on('task', {
    hasSavedRecord (title, ms = 3000) {
      return hasRecordAsync(title, ms)
    }
  })
}

runs in Node in cypress/plugins/index.js

Observe Database Effect

@bahmutov

task completes as soon as the server gets POST from the app and saves record to DB

task checks for wrong title and eventually times out

Node to Browser actions are hard

@bahmutov

Node

Browser

WebDriver.execute

http://webdriver.io/api/protocol/execute.html

browser.execute(script[,argument1,...,argumentN]);

Cy.task

https://on.cypress.io/task

cy.task(name, ...args)

@bahmutov

cy.task
cy.exec
cy.request

(and you only send data, not code)

@bahmutov

Browser to Node is easy

https://applitools.com/blog/cypress-vs-selenium-webdriver-better-or-just-different

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Declarative Syntax

there are no async / awaits or promise chains

Tests should read naturally

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

Puppeteer

@bahmutov

test('My Test', async t => {
    await t
        .setNativeDialogHandler(() => true)
        .click('#populate')
        .click('#submit-button');

    const location = await t.eval(() => window.location);

    await t.expect(location.pathname)
        .eql('/testcafe/example/thank-you.html');
});

TestCafe

@bahmutov

it('changes the URL when "awesome" is clicked', () => {
  const user = cy

  user.visit('/my/resource/path')

  user.get('.awesome-selector')
    .click()

  user.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress is like a real user

Kent C Dodds https://testingjavascript.com/

@bahmutov

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress: all commands are in a queue

visit
get
click

https://on.cypress.io/introduction-to-cypress

@bahmutov

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress: all commands are in a queue

visit
get
click
url
should

https://on.cypress.io/introduction-to-cypress

@bahmutov

Single Command Queue

visit
get
click
url
should

Like a SINGLE user driving the browser

https://on.cypress.io/introduction-to-cypress

@bahmutov

Single Command Queue

visit
get
click
url
should

Deterministic and repeatable

https://on.cypress.io/introduction-to-cypress

@bahmutov

Single Command Queue

visit
get
click
url
should

https://on.cypress.io/introduction-to-cypress

Lazy

@bahmutov

Single Command Queue

visit
get
click
url
should

Lazy

https://on.cypress.io/introduction-to-cypress

@bahmutov

Single Command Queue

visit
get
click
url
should

https://on.cypress.io/introduction-to-cypress

Lazy

@bahmutov

Single Command Queue

visit
get
click
url
should

https://on.cypress.io/introduction-to-cypress

Lazy

@bahmutov

Single Command Queue

visit
get
click
url
should

https://on.cypress.io/introduction-to-cypress

Lazy

@bahmutov

Single Command Queue

visit
get
click
url
should

Commands can be retried (unlike promises)

https://on.cypress.io/introduction-to-cypress

@bahmutov

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  const url = await cy.url()
  url.should('include', 
             '/my/resource/path#awesomeness')
})

NO!

Promises and async / await are eager single try constructs that do not work (well enough) for E2E tests

https://on.cypress.io/introduction-to-cypress

@bahmutov

visit
get
click
url
should

Test starts

water starts flowing

can water reach finish?

@bahmutov

visit
get
click
url
should

Test starts

events starts flowing

can events reach subscribe?

Reactive stream

range(0, 10).pipe(
  filter(x => x % 2 === 0),
  map(x => x + x),
  scan((acc, x) => acc + x, 0)
)
.subscribe(x => 
    console.log(x))

https://rxjs-dev.firebaseapp.com/

@bahmutov

visit
get
click
url
should

writing Cypress test is like writing a single reactive stream

@bahmutov

Cypress vs X

@bahmutov

People testing with Selenium / WebDriver

Cypress users?

No E2E tests

Make The Test Runner Yours

@bahmutov

https://hyper.is/

@bahmutov

Huge list at https://github.com/bnb/awesome-hyper

https://www.npmjs.com/package/hyperpower

@bahmutov

@bahmutov

before(() => {
  console.log('parent.window.document.body is', 
    parent.window.document.body)
})
before(() => {
  const $head = Cypress.$(parent.window.document.head)
  const css = '...' // new style
  $head.append(`<style 
    type="text/css" id="cypress-dark">
    ${css}
  </style>`)
})

your test code

your test code

@bahmutov

https://www.cypress.io/blog/2018/10/11/halloween-theme/

@bahmutov

@bahmutov

End-to-end testing is hard -

but it doesn't have to be

Gleb Bahmutov, PhD

VP of Engineering, Cypress.io

Thank you 👏

https://slides.com/bahmutov/reactive-conf

https://github.com/cypress-io/cypress ✨

video at: https://www.youtube.com/watch?v=swpz0H0u13k