Gleb Bahmutov, PhD

VP of Engineering, Cypress.io

Cypress Test Runner

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

EveryScape

virtual tours

MathWorks

MatLab on the web

Kensho

finance dashboards

15 people. Atlanta, Philly, Boston, LA, Chicago, NYC

Fast, easy and reliable testing for anything that runs in a browser

Why is all software broken?

Will Klein

Quality software behaves the way users expect it to behave

We going to need some tests

E2E

integration

unit

Smallest pieces

Testing Pyramid △

Unit tests pass...

E2E

integration

unit

Component

E2E

integration

unit

Website / API

E2E

integration

unit

Really important to users

Really important to developers

$ npm install -D cypress
it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

Cypress demo

  • Typical test
  • Failing test
  • Recording tests
  • CI setup
  • Cypress dashboard

DOM

storage

location

cookies

Cypress acts as a proxy for your app

specs

app

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Declarative Syntax

there are no async / awaits or promise chains

Tests should read naturally

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

Puppeteer

test('My Test', async t => {
    await t
        .setNativeDialogHandler(() => true)
        .click('#populate')
        .click('#submit-button');

    const location = await t.eval(() => window.location);

    await t.expect(location.pathname)
        .eql('/testcafe/example/thank-you.html');
});

TestCafe

it('changes the URL when "awesome" is clicked', () => {
  const user = cy

  user.visit('/my/resource/path')

  user.get('.awesome-selector')
    .click()

  user.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress is like a real user

Dear user, 

  1. open url localhost:3000/my/resource/path
  2. click on button "foo"
  3. check if url includes /my/resource/path#awesomeness 

more details: "End-to-end Testing Is Hard - But It Doesn't Have to Be​" ReactiveConf 2018 https://www.youtube.com/watch?v=swpz0H0u13k

But how does it FEEL?

When I test with Cypress

DOM

Network

storage

If you can write E2E tests in a framework-agnostic way

You can replace framework X with Y

(without breaking things)

But how is this different from tool X?

Hint: browsers are complicated

Node to Browser actions are hard

Node

Browser

Node

Cy backend

cy.task(name, ...args)
on('task', {
  name: (...args) => ...
})

Browser to Node is easy

cy.task
cy.exec
cy.request

(and you only send data, not code)

Browser to Node is easy

cy.task

Browser to Node is easy

Tutorials

Api

Examples

Video course

Full workshop

  • "cypress open" - GUI interactive mode

  • "cypress run" - headless mode

Cypress CLI has 2 main commands

full video of the run, screenshots of every  failure

  • Every CI should be good

  • Or use a Docker image we provide

Running E2E on CI

Other Demos

many presentations and videos about Cypress

Paid Features 💵

Paid Features 💵: artifacts

test output, video, screenshots

Paid Features 💵 - load balancing

cypress run --record
cypress run --record --group single
cypress run --record --group parallel --parallel

Most CIs should just work 🙏

Paid Features 💵 - load balancing

cypress-dashboard parallelization

# machines run duration time savings
1 22:50 ~
2 11:47 48%
3 7:51 65%
4 5:56 74%
6 3:50 83%
8 3:00 87%
10 2:19 90%

cypress-dashboard parallelization

10 machines = 10x speed up

Cool Things You Can do With Cypress

Test components from frameworks

E2E

integration

unit

Cypress

$ npm install -D cypress cypress-vue-unit-test
const mountVue = require('cypress-vue-unit-test')
describe('My Vue', () => {
  beforeEach(mountVue(/* my Vue code */, /* options */))
  it('renders', () => {
    // Any Cypress command
    // Cypress.vue is the mounted component reference
  })
})

Vue component test demo with Cypress

Interact, inspect, use

Test components from these frameworks with ease

E2E

integration

unit

Web Application

Jest

Cypress

E2E

integration

unit

Web Application

Jest

Cypress

API tests

const request = require('supertest')
const app = require(...)
request(app)
  .get('/user')
  .expect('Content-Type', /json/)
  .expect('Content-Length', '15')
  .expect(200, {
    name: 'Joe',
    age: 33
  })

Replace API tests

it('returns JSON', () => {
  cy.request('http://localhost:3000/todos')
    .its('headers')
    .its('content-type')
    .should('include', 'application/json')
})

With Cypress API tests

and gain test UI

and gain:

  • fixtures

  • stubbing

  • custom commands

  • longer tests

  • mixed UI / API tests

Code Coverage

  • Instrument the code

  • Run the tests

  • See if we reached every line of code

Coverage is hard

Code coverage

is tricky

const isEmail = (s) =>
  /^\w+@\w+\.\w{3,4}$/.test(s)

// 1 test = 100% code coverage
​(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

Code coverage

vs

Data coverage

mocha -r data-cover spec.js

I don't think code coverage is useful for end-to-end tests

You cannot test every part of your car by driving around

user interface

application code

vendor code

polyfills

there is probably a lot of app code unreachable through the UI alone

beforeEach(() => {
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

What does this test cover?

Cypress.Commands.overwrite('type', 
  (type, $el, text, options) => {

  rememberSelector($el)

  return type($el, text, options)
})

Track "cy.type" elements

Highlight tested element

Elements NOT covered by the test

beforeEach(() => {
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
    .first().find(':checkbox').check()

  cy.contains('.filters a', 'Active').click()
  cy.url().should('include', 'active')

  cy.contains('.filters a', 'Completed').click()
  cy.url().should('include', 'completed')

  cy.contains('.filters a', 'All').click()
  cy.url().should('include', '#/')
})

Extend test to cover more elements

The test did not cover "Clear completed" button

Problem: only "check" this box,

but not "uncheck"

Tests should cover all important states of the app, not UI

UI = f (state)

tests = g (UI)

tests = h (state)

import { Machine } from 'xstate';

const lightMachine = Machine({
  id: 'light',
  initial: 'green',
  states: {
    green: {
      on: {
        TIMER: 'yellow',
      }
    },
    yellow: {
      on: {
        TIMER: 'red',
      }
    },
    red: {
      on: {
        TIMER: 'green',
      }
    }
  }
});
const ticTacToeMachine = Machine(
  {
    initial: "playing",
    states: {
      playing: {
        on: {
          "": [
            { target: "winner", cond: "checkWin" },
            { target: "draw", cond: "checkDraw" }
          ],
          PLAY: [
            {
              target: "playing",
              cond: "isValidMove",
              actions: "updateBoard"
            }
          ]
        }
      },
      winner: {
        onEntry: "setWinner"
      },
      draw: {
        type: "final"
      }
    }
  }
  ...
}
const {
  getShortestValuePaths
} = require('xstate/lib/graph')
const ticTacToeMachine = require('./machine')
const shortestValuePaths = getShortestValuePaths(ticTacToeMachine)
const winnerXPath = filterWinnerX(shortestValuePaths)
const winnerOPath = filterWinnerY(shortestValuePaths)
const draws = filterDraw(shortestValuePaths)

tests = h (state)

it('player X wins', () => {
  play(winnerXPath)
  cy.contains('h2', 'Player X wins!')
})

it('player O wins', () => {
  play(winnerOPath)
  cy.contains('h2', 'Player O wins!')
})

draws.forEach((draw, k) => {
  it(`plays to a draw ${k}`, () => {
    const drawPath = shortestValuePaths[draw]
    play(drawPath)
    cy.contains('h2', 'Draw')
  })
})

tests = h (state)

Cypress running autogenerated tests

tests

code

coverage

data

coverage

element

coverage

state

coverage

Coverage

Why developers love Cypress

  • Fast, flake-free

  • GUI, time travel

  • Test recording

  • Documentation

Why developers love Cypress

  • Fast, flake-free

  • GUI, time travel

  • Test recording

  • Documentation

100% FREE & OSS

Coming soon

  • Retries / flake factor

  • Cross-browser

  • Full network stubbing

  • so many more ideas ...

IE11

Gleb Bahmutov, PhD

VP of Engineering

Thank you

@cypress_io

@bahmutov

slides.com/bahmutov/cypress-sf-js

github.com/cypress-io/cypress ⭐️

Cypress Test Runner: SF JS Meetup

By Gleb Bahmutov

Cypress Test Runner: SF JS Meetup

Testing is hard. End-to-end testing is really hard. But have you ever wondered if maybe our tools are the ones to blame? What if we could completely rethink testing tools to be fast, useful and effective? Let me show you how to quickly test any web application using Cypress.io - a modern open source test runner designed to make you more productive. Don't believe the hype? Come see a Cypress demo for yourself and prepare to be amazed.

  • 227
Loading comments...

More from Gleb Bahmutov