I See What is Going On

Visual Testing for Your Components

Gleb Bahmutov

VP of Engineering

Cypress.io 

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

@bahmutov

survival is possible. but we need to act now

  • change your life
  • join an organization

@bahmutov

https://github.com/bahmutov/sudoku

fork of https://github.com/raravi/sudoku live https://sudoku-raravi.now.sh/

https://github.com/bahmutov/sudoku

fork of https://github.com/raravi/sudoku live https://sudoku-raravi.now.sh/

Built from React Components

import React from 'react'
import { render } from 'react-dom'
import { App } from './App'

render(<App />, document.getElementById('root'))

index.js

import React from 'react'
import { Game } from './Game'
import './App.css'
import { SudokuProvider } from './context/SudokuContext'

export const App = () => {
  return (
    <SudokuProvider>
      <Game />
    </SudokuProvider>
  )
}

App.js

Top level component App

import React, { useState, useEffect } from 'react'
import moment from 'moment'
import { Header } from './components/layout/Header'
import { GameSection } from './components/layout/GameSection'
import { StatusSection } from './components/layout/StatusSection'
import { Footer } from './components/layout/Footer'
import { getUniqueSudoku } from './solver/UniqueSudoku'
import { useSudokuContext } from './context/SudokuContext'

export const Game = () => {
  ...
}

Game.js

Game component

return (
  <>
    <div className={overlay?"container blur":"container"}>
      <Header onClick={onClickNewGame}/>
      <div className="innercontainer">
        <GameSection
          onClick={(indexOfArray) => onClickCell(indexOfArray)}
        />
        <StatusSection
          onClickNumber={(number) => onClickNumber(number)}
          onChange={(e) => onChangeDifficulty(e)}
          onClickUndo={onClickUndo}
          onClickErase={onClickErase}
          onClickHint={onClickHint}
          onClickMistakesMode={onClickMistakesMode}
          onClickFastMode={onClickFastMode}
        />
      </div>
      <Footer />
    </div>
  </>
)

Game.js

Game component

Component Inputs

import React from 'react';
import { useSudokuContext } from '../context/SudokuContext';

/**
 * React component for the Number Selector in the Status Section.
 */
export const Numbers = (props) => {
  let { numberSelected } = useSudokuContext();
  return (
    <div className="status__numbers">
      {
        [1, 2, 3, 4, 5, 6, 7, 8, 9].map((number) => {
          if (numberSelected === number.toString()) {
            return (
              <div className="status__number status__number--selected"
                key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          } else {
            return (
              <div className="status__number" key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          }
        })
      }
    </div>
  )
}

Numbers.js

<Numbers onClickNumber={(number) => props.onClickNumber(number)} />

StatusSection.js

<Numbers .../>

props

context

user clicks

DOM

prop calls

Video Moving around the testing pyramid 

slides https://slides.com/bahmutov/assertjs

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

yarn add -D cypress cypress-react-unit-test

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

// cypress/support/index.js
require('cypress-react-unit-test/support')
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  return config
}
// cypress.json
{
  "experimentalComponentTesting": true,
  "componentFolder": "src"
}

@bahmutov

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

apply global styles

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

set the right structure

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

click a number

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

Does it look right?

If I change this CSS (or class name or layout) just a little bit ...

@bahmutov

desktop

tablet

mobile

Does it look the same?

If I change this CSS (or class name or layout) just a little bit ...

Image Diff Plugin

https://github.com/palmerhq/cypress-image-snapshot

yarn add -D cypress-image-snapshot
// cypress/support/index.js
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
  return config
}
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

confirm the DOM has been updated

take image snapshot

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
    cy.get('.status__numbers')
      .matchImageSnapshot('numbers-selected')
  })
})

Commit this image into source control

Let's change CSS

.status__number {
-   padding: 12px 0;
+   padding: 10px 0;
}

App.css

baseline image

new image

difference

cypress/snapshots/Numbers.spec.js/__diff_output__/numbers-selected.diff.png

Deterministic Data: the Clock

import { App } from './App'
it('shows the timer', () => {
  mount(<App />)
  // the timer starts at zero, so this is probably ok
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
})

What about the clock after 10 minutes?

What if the snapshot "catches" clock in transition?

Deterministic Data: the Clock

import { App } from './App'
it('shows the timer', () => {
  cy.clock()
  mount(<App />)
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
  cy.tick(700 * 1000)
  cy.contains('.status__time', '11:40')
    .matchImageSnapshot('timer-passed')
})

https://on.cypress.io/clock

@bahmutov

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

@bahmutov

Why not the entire game?

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

?

<App />

<Game />

<Header />

<GameSection />

<StatusSection />

<Footer />

<Timer />

<Difficulty />

<Numbers />

tested

tested

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

?

<App />

<Game />

<Header />

<GameSection />

<StatusSection />

<Footer />

<Timer />

<Difficulty />

<Numbers />

Deterministic Data: Mocks

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

App.spec.js

?

<App />

<Game />

<Header />

<GameSection />

<StatusSection />

<Footer />

<Timer />

<Difficulty />

<Numbers />

// App.js uses Game.js
// Game.js
import { getUniqueSudoku } from './solver/UniqueSudoku'
...
function _createNewGame(e) {
  let [temporaryInitArray, temporarySolvedArray] = getUniqueSudoku(difficulty, e);
  ...
}

// cypress/fixtures/init-array.json
["0", "0", "9", "0", "2", "0", "0", ...]
// cypress/fixtures/solved-array.json
["6", "7", "9", "3", "2", "8", "4", ...]

@bahmutov

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('shows the board', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

Deterministic Data: Mocks

mock ES6 import

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('shows the board', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

Deterministic Data: Mocks

it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .matchImageSnapshot('same-game-made-one-move')
})
it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .matchImageSnapshot('same-game-made-one-move')
})

See What The Test Is Doing

Our Progress

  1. Component tests
  2. Visual snapshot tests
  3. Deterministic data
    1. clock control
    2. data mocking
  4. Local workflow
  5. Pull request workflow
yarn cypress open

screenshot 1300x600
yarn cypress run

screenshot 650x300

interactive mode

headless mode

Local Workflow

@bahmutov

Pixel density on Mac ...

🔥

Local Workflow

// cypress/support/index.js
import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
if (Cypress.config('isInteractive')) {
  Cypress.Commands.add('matchImageSnapshot', () => {
    cy.log('Skipping snapshot 👀')
  })
} else {
  addMatchImageSnapshotCommand()
}
require('cypress-react-unit-test/support')

Do not take snapshots in interactive mode

@bahmutov

Local Workflow

// cypress/support/index.js
import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
if (Cypress.config('isInteractive')) {
  Cypress.Commands.add('matchImageSnapshot', () => {
    cy.log('Skipping snapshot 👀')
  })
} else {
  addMatchImageSnapshotCommand()
}
require('cypress-react-unit-test/support')

Do not take snapshots in interactive mode

@bahmutov

"yarn cypress run"

  • add a new snapshot

Local Workflow

"CYPRESS_updateSnapshots=true yarn cypress run"

  • update existing snapshot

Run on CI

Take image snapshots using exactly the same environment and browser as CI

Mac cypress run

Linux cypress run

difference

@bahmutov

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

"yarn run docker:run"

  • add a new snapshot
  • update existing snapshot

Use Docker Everywhere

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0" 
  }
}

package.json

jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.13.0-chrome80-ff74

CI config

exactly the same dependencies

Pull Request Workflow

  • run all tests, saving all DIFF images
  • do NOT fail tests on image differences
CYPRESS_failOnSnapshotDiff=false

Pull Request Workflow

  • do NOT fail tests on image differences
CYPRESS_failOnSnapshotDiff=false
  • report # of visual differences (if any)
async function getVisualDiffs() {
  return globby('cypress/snapshots/**/__diff_output__/**.png')
}
getVisualDiffs().then(list => {
  return setGitHubCommitStatus(list.length, envOptions)
}).catch(onError)
  • run all tests, saving all DIFF images

Pull Request Workflow

https://github.com/bahmutov/sudoku

Summary

@bahmutov

Summary

👍 cypress-image-snapshot is good and free

👎 a lot of effort to render images, store them, set up review workflow

Consider 3rd party paid service: Applitools, Happo.io, Percy.io

@bahmutov

Thank you 👏

Want to learn more?

https://github.com/bahmutov/sudoku#videos 

is 1 hour of free videos showing every step in detail

 

Cypress visual testing guide is at https://on.cypress.io/visual-testing

https://slides.com/bahmutov/i-see-what-is-going-on

@bahmutov

I See What is Going On

By Gleb Bahmutov

I See What is Going On

Visual Testing for Your Components. Many modern web frameworks use components as their building blocks. In this talk, I will show component testing using Cypress - and I mean "show" literally. You will see the component running inside a real browser, you will interact with the component like a real user, and you can debug the component using the real browser's DevTools. Finally, I will go through the current open source and commercial options for visually testing the component's styles and rendering. Read blog post https://glebbahmutov.com/blog/open-source-visual-testing-of-components/ Video at https://www.youtube.com/watch?v=00BNExlJUU8

  • 15,399

More from Gleb Bahmutov