Visual Testing &

Code Coverage

Gleb Bahmutov

Cypress.io

@bahmutov

these slides

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

24 people. Atlanta, Philly, Boston, NYC, Nashville,

Myanmar, Colombia, Ukraine

Fast, easy and reliable testing for anything that runs in a browser

Write some E2E tests

npm i -D cypress
docker run cypress/included

or download Cypress application

1

or use Docker image

2

Write E2E test

it('adds and completes todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)

  cy.get('.todo').first().find('.toggle')
    .check()
  cy.get('.todo').first()
    .should('have.class', 'completed')
})
  1. read docs for hours
  2. go through the tutorial*
  3. read more docs
  4. write tests

New Testing Tool

  1. read "Hello World"
  2. copy to your test
  3. write useful tests
  4. gradually improve

New Testing Tool

Our goal: no matter how well you know JavaScript = you are successfully using Cypress

it('changes the URL when "awesome" is clicked', () => {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Declarative Syntax

there are no async / awaits or promise chains

Tests should read naturally

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

Puppeteer

test('My Test', async t => {
    await t
        .setNativeDialogHandler(() => true)
        .click('#populate')
        .click('#submit-button');

    const location = await t.eval(() => window.location);

    await t.expect(location.pathname)
        .eql('/testcafe/example/thank-you.html');
});

TestCafe

it('changes the URL when "awesome" is clicked', () => {
  const user = cy

  user.visit('/my/resource/path')

  user.get('.awesome-selector')
    .click()

  user.url()
    .should('include', 
            '/my/resource/path#awesomeness')
})

Cypress is like a real user

Dear user,

  1. open url localhost:3000/my/resource/path
  2. click on button "foo"
  3. check if url includes /my/resource/path#awesomeness

more details: "End-to-end Testing Is Hard - But It Doesn't Have to Be​" ReactiveConf 2018 https://www.youtube.com/watch?v=swpz0H0u13k

3

Run it on CI

version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          # we need to start the web application
          start: npm start

Tip: use start-server-and-test utility to start server

{
  "scripts": {
    "test": "cypress:run",
    "start": "parcel serve public/index.html",
    "e2e": "start-test 1234"
  }
}

Do I need more end-to-end tests?

Let the code coverage be your guide

But do not let 100% code coverage be the only goal

Istanbul.js - code instrumentation library

nyc - CLI around Istanbul.js

Code Coverage Demo

@cypress/code-coverage

Cypress.io plugin for saving Istanbul code coverage

import '@cypress/code-coverage/support'

cypress/support/index.js file

module.exports = (on, config) => {
  on('task', require('@cypress/code-coverage/task'))
}

cypress/plugins/index.js file

A single end-to-end test can be very effective at covering a lot of code

The missed lines in the business logic code are targets for more end-to-end tests

Cannot reach the edge case from the end-to-end test

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

cypress/integration/selectors-spec.js

Run unit test from Cypress

and get a perfect 100%

Bonus: full stack code coverage

Send code coverage results to external service

Beyond code coverage

  • user story coverage

  • UI element coverage

  • state machine coverage

Viewport coverage

Does the app work on different screens?

;['macbook-15', 'iphone-6'].forEach(viewport => {
  it(`works on ${viewport}`, () => {
    cy.viewport(viewport)
    cy.visit('/')
    cy.get('.new-todo')
      .type('write code{enter}')
      .type('write tests{enter}')
      .type('deploy{enter}')
    cy.get('.todo').should('have.length', 3)

    cy.get('.todo')
      .first()
      .find('.toggle')
      .check()
    cy.get('.todo')
      .first()
      .should('have.class', 'completed')

    cy.get('.clear-completed').click()
    cy.get('.todo').should('have.length', 2)
  })
})
// import 'todomvc-app-css/index.css'

functional tests still pass!

🚫

don't try assert visual look

Visual testing

=

?

baseline image

new image

visual diff

Visual Testing Challenges

🙁 Hard to test complex app states

     😍 — but not with Cypress!

😵 Performance

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

Do-It-Yourself vs 3rd party service

Disclaimer: PhD in Computer Vision and Image Processing

Visual Testing Demo

  • Percy.io plugin and commands

  • Consistent application state

  • GitHub pull request workflow

Pull request workflow

runs first

added async

Cypress GitHub integration is coming soon

https://github.com/cypress-io/cypress/issues/981

Functional tests

Visual testing

Code coverage

Gleb Bahmutov

Cypress.io

@bahmutov

Thank you