No More CSS Regrets, ... I mean Regressions

Gleb Bahmutov

@bahmutov

Sr. Director of Engineering

MercariUS 

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Agenda

  • Functional vs Visual testing

  • Why visual testing?

  • Visual testing principles

  • Visual testing services

  • Do-It-Yourself visual testing

it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})
$ npx cypress open

Would you sign in?

Can you sign in?

Would you click Buy?

  • Break user trust
  • Loss of business 📉
  • Hard to test using functional assertions
cy.get('.buy')
  .should('be.visible')
  .and('have.text', 'Buy Now')
cy.get('.secure-note')
  .should('be.visible')
  .and('have.text', 'Secure transaction')

Bugs like these:

  • Break user trust
  • Loss of business 📉
  • Hard to test using functional assertions

every element, every style and layout property...

Bugs like these:

How to test the label element and its position?

Manual testing does not scale

Functional end-to-end tests are not very helpful 

// import 'todomvc-app-css/index.css'

functional tests still pass!

🚫

don't try assert visual look

Visual testing: the idea

  • Render the page or an element into an image

  • Compare the image to the previously rendered and approved image (the baseline)

Visual testing

=

baseline image

new image

visual diff

Visual testing

=

baseline image

new image

visual diff

👤 rejects the difference

Test fails ❌

Visual testing

=

baseline image

new image

visual diff

👤 accepts the difference

Test passes ✅

Visual testing

baseline image

Accepted image becomes the new baseline image

Does it look the same?

👩‍💻 🛑 🕰

🤖 ✅ ⏱

Visual Testing:

How many images?

  • Full app screenshot at each major step of every user flow
  • At each supported resolution
  • Plus the different states of components

Example: Sudoku with responsive design = 15 images

https://github.com/bahmutov/sudoku-qafest

App at different stages of the game

Tablet resolution

Components

Visual Testing Challenges

🙁 Hard to test complex app states with dynamic data, clock

     😍 — but not with Cypress!

😵 Performance: 100s of images

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

Decision point:

Do-It-Yourself vs 3rd party service

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

Expensive 💰.

Disclaimer: PhD in Computer Vision and Image Processing

Recommended ✅

it('Loads the homepage', function() {
  cy.visit('/')

  // Take a snapshot for visual diffing
  
  // https://docs.percy.io/docs/cypress
  cy.percySnapshot()
  
  // https://applitools.com/tutorials/cypress.html
  cy.eyesCheckWindow()
  
  // https://docs.happo.io/docs/cypress
  cy.get('.app').happoScreenshot()
})

3rd Party Service

3rd Party Service: webinars

"Visual Testing with Cypress and Percy"

"Cypress and Happo"

"Component Testing With Cypress and Applitools"

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

The app

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

The app

An element

Image Diff Plugin

yarn add -D cypress-image-snapshot
// cypress/support/index.js
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
  return config
}
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

confirm the DOM has been updated

take image snapshot

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

Commit this image into source control

Let's change CSS

.status__number {
-   padding: 12px 0;
+   padding: 10px 0;
}

App.css

baseline image

new image

difference

cypress/snapshots/Numbers.spec.js/__diff_output__/numbers-selected.diff.png

baseline image

new image

difference

App visual diff due to difference board

Speed matters

Do-It-Yourself: Images do not match...

Use local Docker container to run Cypress and render consistent images

Mac vs Linux font rendering

Ignore differences < N%

OR

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

"yarn run docker:run"

  • add a new snapshot
  • update existing snapshot

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.13.0-chrome80-ff74

CI config

exactly the same dependencies

Alternative: Send DOM Snapshot only

it('saves the page', () => {
  // normal Cypress commands
  cy.contains('.some-selector', 'some text')
    .should('be.visible')
    // when the app has reached the desired state
    // save the page in the folder "page"
    .savePage('page')
})

Storing Images

Comparing and approval workflows

Visual Regression Tracker

  • Takes local screenshots via cy.screenshot command ⚠️
    • does not solve the consistent rendering problem
  • Stand-alone Docker compose project
  • Keeps baseline images and info in Postgres DB
  • Image review and approval web interface

Do-it-yourself image comparison and management

/// <reference types="cypress" />

describe('Sudoku', () => {
  it('shows the initial game', () => {
    cy.vrtStart()

    // stop the game clock
    cy.clock()

    // load the initial and solved game boards
    cy.fixture('init-array.json').then((initialBoard) => {
      cy.fixture('solved-array.json').then((solvedBoard) => {
        cy.visit('/', {
          onBeforeLoad(win) {
            win.__initArray = initialBoard
            win.__solvedArray = solvedBoard
          },
        })
      })
    })

    // the board fixture has 45 filled cells at the start
    cy.get('.game__cell--filled').should('have.length', 45)
    cy.contains('.status__time', '00:00').should('be.visible')
    cy.contains('.status__difficulty-select', 'Easy').should('be.visible')

    cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
    cy.vrtStop()
  })
})

Typical End-to-end Visual Test

controls the randomness and the game clock

Cypress Test Runner: first run

VRT dashboard on first run

"docker-compose up"

The screenshot matches the baseline

Run Cypress tests again

  onBeforeLoad(win) {
+   initialBoard[2] = 8
    win.__initArray = initialBoard
    win.__solvedArray = solvedBoard
  },
  
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
  cy.vrtTrack('sudoku')

if the test or app changes

Image approval workflow

Bend the Rules: Mock modules in the JS bundles

import React from 'react'
import { mockInBundle } from 'mock-in-bundle'
import initialBoard from '../fixtures/init-array.json'
import solvedBoard from '../fixtures/solved-array.json'

describe('Sudoku', () => {
  it('mocks Timer and UniqueSudoku', () => {
    mockInBundle(
      {
        'Timer.js': {
          Timer: () => <div className="status__time">TIME</div>,
        },
        'UniqueSudoku.js': {
          getUniqueSudoku: () => [initialBoard, solvedBoard],
        },
      },
      'main.chunk.js',
    )
    cy.visit('/')
    cy.contains('.status__time', 'TIME').should('be.visible')
    // take visual snapshot
  })
})

Visual Flake

Visual Flake

Visual Flake

Best Practice

// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')

Best Practice

Same data

Same exact environment

Consistent image for comparison

Same HTML + CSS

Take and compare screenshots during the tests and fail the tests if images do not match

Post visual diffing results as a separate status check on PR

OR

Pull Request Workflow

Learn more:

No More CSS Regrets, ... I mean Regressions

Gleb Bahmutov

@bahmutov

Thank you 👏