Visual Testing using Cypress

Gleb Bahmutov

@bahmutov

Distinguished Engineer

Cypress.io 

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

(these slides)

50 people. Atlanta, Philly, Boston, NYC, the world

Fast, easy and reliable testing for anything that runs in a browser

$ npm install -D cypress
it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})
$ npx cypress open

Agenda

  • Why visual testing?

  • Visual testing principles

  • Visual testing services

  • Do-It-Yourself visual testing

  • Visual testing for canvas elements

Would you sign in?

Can you sign in?

Would you click Buy?

  • Break user trust
  • Loss of business
  • Hard to test using functional assertions
cy.get('.buy')
  .should('be.visible')
  .and('have.text', 'Buy Now')
cy.get('.secure-note')
  .should('be.visible')
  .and('have.text', 'Secure transaction')

every element, every style and layout property...

Bugs like these:

Manual testing does not scale

desktop

tablet

mobile

Look at every application flow at every resolution

Functional end-to-end tests are not very helpful 

// import 'todomvc-app-css/index.css'

functional tests still pass!

🚫

don't try assert visual look

Visual testing: the idea

  • Render the page or an element into an image

  • Compare the image to the previously rendered and approved image (the baseline)

Visual testing

=

baseline image

new image

visual diff

Visual testing

=

baseline image

new image

visual diff

👤 rejects the difference

Test fails ❌

Visual testing

=

baseline image

new image

visual diff

👤 accepts the difference

Test passes ✅

Visual testing

baseline image

Accepted image becomes the new baseline image

Does it look the same?

👩‍💻 🛑 🕰

🤖 ✅ ⏱

Visual Testing:

How many images?

  • Full app screenshot at each major step of every user flow
  • At each supported resolution
  • Plus the different states of components

Example: Sudoku with responsive design = 15 images

https://github.com/bahmutov/sudoku-qafest

Tablet resolution

Components

Visual Testing Challenges

🙁 Hard to test complex app states with dynamic data, clock

     😍 — but not with Cypress!

😵 Performance: 100s of images

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

Do-It-Yourself vs 3rd party service

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

Expensive 💰.

Disclaimer: PhD in Computer Vision and Image Processing

Recommended ✅

it('Loads the homepage', function() {
  cy.visit('/')

  // Take a snapshot for visual diffing
  
  // https://docs.percy.io/docs/cypress
  cy.percySnapshot()
  
  // https://applitools.com/tutorials/cypress.html
  cy.eyesCheckWindow()
  
  // https://docs.happo.io/docs/cypress
  cy.get('.app').happoScreenshot()
})

3rd Party Service

3rd Party Service: webinars

"Visual Testing with Cypress and Percy"

"Cypress and Happo"

"Component Testing With Cypress and Applitools"

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

The app

An element

Do-It-Yourself

Mac vs Linux font rendering

Do-It-Yourself

Extra complexity 🤯

Visual Regression Tracker

  • Takes local screenshots via cy.screenshot command ⚠️
    • does not solve the consistent rendering problem
  • Stand-alone Docker compose project
  • Keeps baseline images and info in Postgres DB
  • Image review and approval web interface

Do-it-yourself image comparison and management

/// <reference types="cypress" />

describe('Sudoku', () => {
  it('shows the initial game', () => {
    cy.vrtStart()

    // stop the game clock
    cy.clock()

    // load the initial and solved game boards
    cy.fixture('init-array.json').then((initialBoard) => {
      cy.fixture('solved-array.json').then((solvedBoard) => {
        cy.visit('/', {
          onBeforeLoad(win) {
            win.__initArray = initialBoard
            win.__solvedArray = solvedBoard
          },
        })
      })
    })

    // the board fixture has 45 filled cells at the start
    cy.get('.game__cell--filled').should('have.length', 45)
    cy.contains('.status__time', '00:00').should('be.visible')
    cy.contains('.status__difficulty-select', 'Easy').should('be.visible')

    cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
    cy.vrtStop()
  })
})

Typical End-to-end Visual Test

controls the randomness and the game clock

Cypress Test Runner: first run

VRT dashboard on first run

"docker-compose up"

The screenshot matches the baseline

Run Cypress tests again

  onBeforeLoad(win) {
+   initialBoard[2] = 8
    win.__initArray = initialBoard
    win.__solvedArray = solvedBoard
  },
  
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
  cy.vrtTrack('sudoku')

if the test or app changes

Image approval workflow

Visual Flake

Visual Flake

Visual Flake

Best Practice

// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')

Visual Testing for Canvas

Image comparison with retries

"How do we know the canvas has finished rendering?"

How does it know when to grab the canvas element to compare?

// this test passes but only because it waits a long time
// for the animation to be finished before taking the image
it('smiles broadly with wait', () => {
  cy.visit('/smile')
  cy.wait(4000) // ANTI-PATTERN: HARDCODED 4s WAIT

  downloadPng('smile.png').then((filename) => {
    cy.log(`saved ${filename}`)
    cy.task('compare', { filename })
      .should('deep.equal', { match: true })
  })
})

If the test waits just 1 second

then the images do not match

cy.get('.todo-list li')     // command
  .should('have.length', 2) // assertion

Visual Retry-ability

  1. Take screenshot
  2. Compare to the baseline image
  3. If matches ✅
  4. Else
    1. if out of time ❌
    2. Else go back to #1

Visual Retry-ability

import { recurse } from 'cypress-recurse'
it('smiles broadly', () => {
  cy.visit('/smile')
  // notice: no cy.wait!
  recurse(
    () => { // Cypress commands to run
      return downloadPng('smile.png').then((filename) => {
        cy.log(`saved ${filename}`)
        return cy.task('compare', { filename })
      })
    },
    ({ match }) => match, // predicate
  )
})

https://github.com/bahmutov/cypress-recurse

A way to re-run Cypress commands until a predicate function returns true

Visual Retry-ability

import { looksTheSame } from './utils'
it('looks the same', () => {
  cy.visit('/smile')
  looksTheSame('smile.png')
})

Factor out the common code

Wait for Canvas to be Static

// pixelmatch NPM library runs in the browser
import pixelmatch from 'pixelmatch'

cy.get('canvas').then(($canvas) => {
  const ctx1 = $canvas[0].getContext('2d')
  const width = $canvas[0].width
  const height = $canvas[0].height
  const img1 = ctx1.getImageData(0, 0, width, height)

  cy.wait(3000)
  // the image has finished animation, maybe
  cy.get('canvas').then(($canvas) => {
    const ctx2 = $canvas[0].getContext('2d')
    const img2 = ctx2.getImageData(0, 0, width, height)

    const diff = ctx2.createImageData(width, height)
    const number = pixelmatch(img1.data, img2.data, diff.data, width, height)
    // if number === 0
    //   images are the same
  })
})

Wait for Canvas to be Static

recurse(
  () => {
    return cy.get(selector, noLog).then(($canvas) => {
      // got img2
      const number = pixelmatch(img1, img2, ...)

      // for next comparison, use the new image
      // as the base - this way we can get to the end
      // of any animation
      img1 = img2

      return number
    })
  },
  (numberOfDifferentPixels) => numberOfDifferentPixels < 10,
  {
    // by default uses the default command timeout
    log: (numberOfDifferentPixels) =>
      cy.log(`**${numberOfDifferentPixels}** diff pixels`),
    delay,
  },
)

Wait for Canvas to be Static

it('can wait for canvas to become static', () => {
  cy.visit('/smile')
  ensureCanvasStatic('canvas')
  // after this it is pretty much guaranteed to
  // immediately pass the image diffing on the 1st try
  looksTheSame('smile.png', false)
})

Wait for Canvas to be Static

it('can wait for canvas to become static', () => {
  cy.visit('/smile')
  ensureCanvasStatic('canvas')
  // after this it is pretty much guaranteed to
  // immediately pass the image diffing on the 1st try
  looksTheSame('smile.png', false)
})

Example: Canvas charts

import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
  const log = false

  cy.visit('/bar')
  looksTheSame('bar-chart.png', log)

  cy.contains('button', 'Add Dataset').click()
  looksTheSame('bar-chart-added-dataset.png', log)

  cy.contains('button', 'Add Dataset').click()
  looksTheSame('bar-chart-3-sets.png', log)

  // remove first two data sets
  cy.contains('button', 'Remove Dataset')
    .click().click()
  looksTheSame('bar-chart-third-dataset-only.png', log)
})

Example: Canvas charts

import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
  const log = false

  cy.visit('/bar')
  looksTheSame('bar-chart.png', log)

  cy.contains('button', 'Add Dataset').click()
  looksTheSame('bar-chart-added-dataset.png', log)

  cy.contains('button', 'Add Dataset').click()
  looksTheSame('bar-chart-3-sets.png', log)

  // remove first two data sets
  cy.contains('button', 'Remove Dataset')
    .click().click()
  looksTheSame('bar-chart-third-dataset-only.png', log)
})

Learn more:

Visual Testing using Cypress

Gleb Bahmutov

@bahmutov

Thank you 👏