Visual Testing &

Code Coverage

Gleb Bahmutov


these slides

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

24 people. Atlanta, Philly, Boston, NYC, Nashville,

Myanmar, Colombia, Ukraine

Fast, easy and reliable testing for anything that runs in a browser

Write some E2E tests

npm i -D cypress
docker run cypress/included

or download Cypress application


or use Docker image


Write E2E test

it('adds and completes todos', () => {
    .type('write code{enter}')
    .type('write tests{enter}')
  cy.get('.todo').should('have.length', 3)

    .should('have.class', 'completed')
  1. read docs for hours
  2. go through the tutorial*
  3. read more docs
  4. write tests

New Testing Tool

  1. read "Hello World"
  2. copy to your test
  3. write useful tests
  4. gradually improve

New Testing Tool

Our goal: no matter how well you know JavaScript = you are successfully using Cypress

it('changes the URL when "awesome" is clicked', () => {



Declarative Syntax

there are no async / awaits or promise chains

Tests should read naturally

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('');
  await page.screenshot({path: 'example.png'});

  await browser.close();


test('My Test', async t => {
    await t
        .setNativeDialogHandler(() => true)

    const location = await t.eval(() => window.location);

    await t.expect(location.pathname)


it('changes the URL when "awesome" is clicked', () => {
  const user = cy




Cypress is like a real user

Dear user,

  1. open url localhost:3000/my/resource/path
  2. click on button "foo"
  3. check if url includes /my/resource/path#awesomeness

more details: "End-to-end Testing Is Hard - But It Doesn't Have to Be​" ReactiveConf 2018


Run it on CI

version: 2.1
  cypress: cypress-io/cypress@1
      - cypress/run:
          # we need to start the web application
          start: npm start

Tip: use start-server-and-test utility to start server

  "scripts": {
    "test": "cypress:run",
    "start": "parcel serve public/index.html",
    "e2e": "start-test 1234"

Do I need more end-to-end tests?

Let the code coverage be your guide

But do not let 100% code coverage be the only goal

Istanbul.js - code instrumentation library

nyc - CLI around Istanbul.js

Code Coverage Demo

@cypress/code-coverage plugin for saving Istanbul code coverage

import '@cypress/code-coverage/support'

cypress/support/index.js file

module.exports = (on, config) => {
  on('task', require('@cypress/code-coverage/task'))

cypress/plugins/index.js file

A single end-to-end test can be very effective at covering a lot of code

The missed lines in the business logic code are targets for more end-to-end tests

Cannot reach the edge case from the end-to-end test

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
        todos: [],
        visibilityFilter: 'unknown-filter'


Run unit test from Cypress

and get a perfect 100%

Bonus: full stack code coverage

Send code coverage results to external service

Beyond code coverage

  • user story coverage

  • UI element coverage

  • state machine coverage

Viewport coverage

Does the app work on different screens?

;['macbook-15', 'iphone-6'].forEach(viewport => {
  it(`works on ${viewport}`, () => {
      .type('write code{enter}')
      .type('write tests{enter}')
    cy.get('.todo').should('have.length', 3)

      .should('have.class', 'completed')

    cy.get('.todo').should('have.length', 2)
// import 'todomvc-app-css/index.css'

functional tests still pass!


don't try assert visual look

Visual testing




baseline image

new image

visual diff

Visual Testing Challenges

🙁 Hard to test complex app states

     😍 — but not with Cypress!

😵 Performance

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

Do-It-Yourself vs 3rd party service

Disclaimer: PhD in Computer Vision and Image Processing

Visual Testing Demo

  • plugin and commands

  • Consistent application state

  • GitHub pull request workflow

Pull request workflow

runs first

added async

Cypress GitHub integration is coming soon

Functional tests

Visual testing

Code coverage

Gleb Bahmutov


Thank you

Visual Testing and Code Coverage

By Gleb Bahmutov

Visual Testing and Code Coverage

In this talk, I will use code coverage to guide end-to-end tests writing and visual regression testing to avoid style regressions. I will also explain how asynchronous tests can be made to look synchronous. Presented at BostonJS meetup, June 2019

  • 1,789