I See What is Going On

Visual Testing for Your Components

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

@bahmutov

survival is possible. but we need to act now

change your life
join an organization

@bahmutov

https://github.com/bahmutov/sudoku

fork of https://github.com/raravi/sudoku live https://sudoku-raravi.now.sh/

https://github.com/bahmutov/sudoku

fork of https://github.com/raravi/sudoku live https://sudoku-raravi.now.sh/

Built from React Components

import React from 'react'
import { render } from 'react-dom'
import { App } from './App'

render(<App />, document.getElementById('root'))

index.js

import React from 'react'
import { Game } from './Game'
import './App.css'
import { SudokuProvider } from './context/SudokuContext'

export const App = () => {
  return (
    <SudokuProvider>
      <Game />
    </SudokuProvider>
  )
}

App.js

Top level component App

import React, { useState, useEffect } from 'react'
import moment from 'moment'
import { Header } from './components/layout/Header'
import { GameSection } from './components/layout/GameSection'
import { StatusSection } from './components/layout/StatusSection'
import { Footer } from './components/layout/Footer'
import { getUniqueSudoku } from './solver/UniqueSudoku'
import { useSudokuContext } from './context/SudokuContext'

export const Game = () => {
  ...
}

Game.js

Game component

return (
  <>
    <div className={overlay?"container blur":"container"}>
      <Header onClick={onClickNewGame}/>
      <div className="innercontainer">
        <GameSection
          onClick={(indexOfArray) => onClickCell(indexOfArray)}
        />
        <StatusSection
          onClickNumber={(number) => onClickNumber(number)}
          onChange={(e) => onChangeDifficulty(e)}
          onClickUndo={onClickUndo}
          onClickErase={onClickErase}
          onClickHint={onClickHint}
          onClickMistakesMode={onClickMistakesMode}
          onClickFastMode={onClickFastMode}
        />
      </div>
      <Footer />
    </div>
  </>
)

Game.js

Game component

Component Inputs

import React from 'react';
import { useSudokuContext } from '../context/SudokuContext';

/**
 * React component for the Number Selector in the Status Section.
 */
export const Numbers = (props) => {
  let { numberSelected } = useSudokuContext();
  return (
    <div className="status__numbers">
      {
        [1, 2, 3, 4, 5, 6, 7, 8, 9].map((number) => {
          if (numberSelected === number.toString()) {
            return (
              <div className="status__number status__number--selected"
                key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          } else {
            return (
              <div className="status__number" key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          }
        })
      }
    </div>
  )
}

Numbers.js

<Numbers onClickNumber={(number) => props.onClickNumber(number)} />

StatusSection.js

<Numbers .../>

props

context

user clicks

DOM

prop calls

Video Moving around the testing pyramid

slides https://slides.com/bahmutov/assertjs

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

yarn add -D cypress cypress-react-unit-test

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

// cypress/support/index.js
require('cypress-react-unit-test/support')

// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  return config
}

// cypress.json
{
  "experimentalComponentTesting": true,
  "componentFolder": "src"
}

@bahmutov

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

apply global styles

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

set the right structure

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

click a number

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

Does it look right?

If I change this CSS (or class name or layout) just a little bit ...

@bahmutov

desktop

tablet

mobile

Does it look the same?

If I change this CSS (or class name or layout) just a little bit ...

Image Diff Plugin

https://github.com/palmerhq/cypress-image-snapshot

yarn add -D cypress-image-snapshot

// cypress/support/index.js
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()

// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
  return config
}

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

confirm the DOM has been updated

take image snapshot

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

Commit this image into source control

Let's change CSS

.status__number {
-   padding: 12px 0;
+   padding: 10px 0;
}

App.css

baseline image

new image

difference

cypress/snapshots/Numbers.spec.js/__diff_output__/numbers-selected.diff.png

Deterministic Data: the Clock

import { App } from './App'
it('shows the timer', () => {
  mount(<App />)
  // the timer starts at zero, so this is probably ok
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
})

What about the clock after 10 minutes?

What if the snapshot "catches" clock in transition?

Deterministic Data: the Clock

import { App } from './App'
it('shows the timer', () => {
  cy.clock()
  mount(<App />)
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
  cy.tick(700 * 1000)
  cy.contains('.status__time', '11:40')
    .matchImageSnapshot('timer-passed')
})

https://on.cypress.io/clock

@bahmutov

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

@bahmutov

Why not the entire game?

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

<App />

tested

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

<App />

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

<App />

// App.js uses Game.js
// Game.js
import { getUniqueSudoku } from './solver/UniqueSudoku'
...
function _createNewGame(e) {
  let [temporaryInitArray, temporarySolvedArray] = getUniqueSudoku(difficulty, e);
  ...
}

// cypress/fixtures/init-array.json
["0", "0", "9", "0", "2", "0", "0", ...]

// cypress/fixtures/solved-array.json
["6", "7", "9", "3", "2", "8", "4", ...]

@bahmutov

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('shows the board', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

Deterministic Data: Mocks

mock ES6 import

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('shows the board', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

Deterministic Data: Mocks

it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .matchImageSnapshot('same-game-made-one-move')
})

it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .matchImageSnapshot('same-game-made-one-move')
})

See What The Test Is Doing

Our Progress

Component tests
Visual snapshot tests
Deterministic data
1. clock control
2. data mocking
Local workflow
Pull request workflow

yarn cypress open

screenshot 1300x600

yarn cypress run

screenshot 650x300

interactive mode

headless mode

Local Workflow

@bahmutov

Pixel density on Mac ...

🔥

Local Workflow

// cypress/support/index.js
import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
if (Cypress.config('isInteractive')) {
  Cypress.Commands.add('matchImageSnapshot', () => {
    cy.log('Skipping snapshot 👀')
  })
} else {
  addMatchImageSnapshotCommand()
}
require('cypress-react-unit-test/support')

Do not take snapshots in interactive mode

@bahmutov

Local Workflow

// cypress/support/index.js
import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
if (Cypress.config('isInteractive')) {
  Cypress.Commands.add('matchImageSnapshot', () => {
    cy.log('Skipping snapshot 👀')
  })
} else {
  addMatchImageSnapshotCommand()
}
require('cypress-react-unit-test/support')

Do not take snapshots in interactive mode

@bahmutov

"yarn cypress run"

add a new snapshot

Local Workflow

"CYPRESS_updateSnapshots=true yarn cypress run"

update existing snapshot

Run on CI

Take image snapshots using exactly the same environment and browser as CI

Mac cypress run

Linux cypress run

difference

@bahmutov

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

"yarn run docker:run"

add a new snapshot
update existing snapshot

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.13.0-chrome80-ff74

CI config

exactly the same dependencies

Pull Request Workflow

run all tests, saving all DIFF images

do NOT fail tests on image differences

CYPRESS_failOnSnapshotDiff=false

Pull Request Workflow

do NOT fail tests on image differences

CYPRESS_failOnSnapshotDiff=false

report # of visual differences (if any)

async function getVisualDiffs() {
  return globby('cypress/snapshots/**/__diff_output__/**.png')
}
getVisualDiffs().then(list => {
  return setGitHubCommitStatus(list.length, envOptions)
}).catch(onError)

run all tests, saving all DIFF images

Pull Request Workflow

https://github.com/bahmutov/sudoku

Summary

React component tests using cypress-react-unit-test
Visual snapshotting using cypress-image-snapshot
Deterministic data
Local and Ci workflow

@bahmutov

Summary

👍 cypress-image-snapshot is good and free

👎 a lot of effort to render images, store them, set up review workflow

Consider 3rd party paid service: Applitools, Happo.io, Percy.io

@bahmutov

Thank you 👏

Want to learn more?

https://github.com/bahmutov/sudoku#videos

is 1 hour of free videos showing every step in detail

Cypress visual testing guide is at https://on.cypress.io/visual-testing

https://slides.com/bahmutov/i-see-what-is-going-on

@bahmutov