No More CSS Regrets, ... I mean Regressions

Gleb Bahmutov

@bahmutov

Sr. Director of Engineering

MercariUS 

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Agenda

  • Functional vs Visual testing

  • Why visual testing?

  • Visual testing principles

  • Visual testing services

  • Do-It-Yourself visual testing

it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})
$ npx cypress open

Would you sign in?

Can you sign in?

Would you click Buy?

  • Break user trust
  • Loss of business 📉
  • Hard to test using functional assertions
cy.get('.buy')
  .should('be.visible')
  .and('have.text', 'Buy Now')
cy.get('.secure-note')
  .should('be.visible')
  .and('have.text', 'Secure transaction')

Bugs like these:

  • Break user trust
  • Loss of business 📉
  • Hard to test using functional assertions

every element, every style and layout property...

Bugs like these:

How to test the label element and its position?

Manual testing does not scale

Functional end-to-end tests are not very helpful 

// import 'todomvc-app-css/index.css'

functional tests still pass!

🚫

don't try assert visual look

Visual testing: the idea

  • Render the page or an element into an image

  • Compare the image to the previously rendered and approved image (the baseline)

Visual testing

=

baseline image

new image

visual diff

Visual testing

=

baseline image

new image

visual diff

👤 rejects the difference

Test fails ❌

Visual testing

=

baseline image

new image

visual diff

👤 accepts the difference

Test passes ✅

Visual testing

baseline image

Accepted image becomes the new baseline image

Does it look the same?

👩‍💻 🛑 🕰

🤖 ✅ ⏱

Visual Testing:

How many images?

  • Full app screenshot at each major step of every user flow
  • At each supported resolution
  • Plus the different states of components

Example: Sudoku with responsive design = 15 images

https://github.com/bahmutov/sudoku-qafest

App at different stages of the game

Tablet resolution

Components

Visual Testing Challenges

🙁 Hard to test complex app states with dynamic data, clock

     😍 — but not with Cypress!

😵 Performance: 100s of images

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

Decision point:

Do-It-Yourself vs 3rd party service

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

3rd Party Service

  • Sends DOM and styles to the external cloud

  • Image is rendered in the cloud (consistent, fast)

  • Image is compared with the previously approved baseline image stored in the cloud

  • Status checks are posted asynchronously

Expensive 💰.

Disclaimer: PhD in Computer Vision and Image Processing

Recommended ✅

it('Loads the homepage', function() {
  cy.visit('/')

  // Take a snapshot for visual diffing
  
  // https://docs.percy.io/docs/cypress
  cy.percySnapshot()
  
  // https://applitools.com/tutorials/cypress.html
  cy.eyesCheckWindow()
  
  // https://docs.happo.io/docs/cypress
  cy.get('.app').happoScreenshot()
})

3rd Party Service

3rd Party Service: webinars

"Visual Testing with Cypress and Percy"

"Cypress and Happo"

"Component Testing With Cypress and Applitools"

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

The app

Do-It-Yourself

cy.screenshot({ capture: 'runner' })   // the test runner
cy.screenshot()                        // the application
cy.get('.status').screenshot('status') // an element

The Test Runner

The app

An element

Image Diff Plugin

yarn add -D cypress-image-snapshot
// cypress/support/index.js
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
  return config
}
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

confirm the DOM has been updated

take image snapshot

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

Commit this image into source control

Let's change CSS

.status__number {
-   padding: 12px 0;
+   padding: 10px 0;
}

App.css

baseline image

new image

difference

cypress/snapshots/Numbers.spec.js/__diff_output__/numbers-selected.diff.png

baseline image

new image

difference

App visual diff due to difference board

Speed matters

Do-It-Yourself: Images do not match...

Use local Docker container to run Cypress and render consistent images

Mac vs Linux font rendering

Ignore differences < N%

OR

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

"yarn run docker:run"

  • add a new snapshot
  • update existing snapshot

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.13.0-chrome80-ff74

CI config

exactly the same dependencies

Alternative: Send DOM Snapshot only

it('saves the page', () => {
  // normal Cypress commands
  cy.contains('.some-selector', 'some text')
    .should('be.visible')
    // when the app has reached the desired state
    // save the page in the folder "page"
    .savePage('page')
})

Storing Images

Comparing and approval workflows

Visual Regression Tracker

  • Takes local screenshots via cy.screenshot command ⚠️
    • does not solve the consistent rendering problem
  • Stand-alone Docker compose project
  • Keeps baseline images and info in Postgres DB
  • Image review and approval web interface

Do-it-yourself image comparison and management

/// <reference types="cypress" />

describe('Sudoku', () => {
  it('shows the initial game', () => {
    cy.vrtStart()

    // stop the game clock
    cy.clock()

    // load the initial and solved game boards
    cy.fixture('init-array.json').then((initialBoard) => {
      cy.fixture('solved-array.json').then((solvedBoard) => {
        cy.visit('/', {
          onBeforeLoad(win) {
            win.__initArray = initialBoard
            win.__solvedArray = solvedBoard
          },
        })
      })
    })

    // the board fixture has 45 filled cells at the start
    cy.get('.game__cell--filled').should('have.length', 45)
    cy.contains('.status__time', '00:00').should('be.visible')
    cy.contains('.status__difficulty-select', 'Easy').should('be.visible')

    cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
    cy.vrtStop()
  })
})

Typical End-to-end Visual Test

controls the randomness and the game clock

Cypress Test Runner: first run

VRT dashboard on first run

"docker-compose up"

The screenshot matches the baseline

Run Cypress tests again

  onBeforeLoad(win) {
+   initialBoard[2] = 8
    win.__initArray = initialBoard
    win.__solvedArray = solvedBoard
  },
  
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
  cy.vrtTrack('sudoku')

if the test or app changes

Image approval workflow

Bend the Rules: Mock modules in the JS bundles

import React from 'react'
import { mockInBundle } from 'mock-in-bundle'
import initialBoard from '../fixtures/init-array.json'
import solvedBoard from '../fixtures/solved-array.json'

describe('Sudoku', () => {
  it('mocks Timer and UniqueSudoku', () => {
    mockInBundle(
      {
        'Timer.js': {
          Timer: () => <div className="status__time">TIME</div>,
        },
        'UniqueSudoku.js': {
          getUniqueSudoku: () => [initialBoard, solvedBoard],
        },
      },
      'main.chunk.js',
    )
    cy.visit('/')
    cy.contains('.status__time', 'TIME').should('be.visible')
    // take visual snapshot
  })
})

Visual Flake

Visual Flake

Visual Flake

Best Practice

// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')

Best Practice

Same data

Same exact environment

Consistent image for comparison

Same HTML + CSS

Take and compare screenshots during the tests and fail the tests if images do not match

Post visual diffing results as a separate status check on PR

OR

Pull Request Workflow

Learn more:

No More CSS Regrets, ... I mean Regressions

Gleb Bahmutov

@bahmutov

Thank you 👏

No More CSS Regrets, ... I mean Regressions

By Gleb Bahmutov

No More CSS Regrets, ... I mean Regressions

Visual testing is extremely effective at confirming the application works and looks the way it did previously, and that the new commits have not accidentally broken it. In this presentation, I will show how to do visual testing using the Cypress test runner, both at the end-to-end and at the individual component levels. We will also consider the trade-offs of doing image diffing ourselves vs paying for a 3rd party service. Presented at ConFoo CA 2022

  • 1,723