Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end.
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})
$ npx cypress open
Would you sign in?
Can you sign in?
Would you click Buy?
cy.get('.buy')
.should('be.visible')
.and('have.text', 'Buy Now')
cy.get('.secure-note')
.should('be.visible')
.and('have.text', 'Secure transaction')
every element, every style and layout property...
How to test the label element and its position?
// import 'todomvc-app-css/index.css'
functional tests still pass!
don't try assert visual look
baseline image
new image
visual diff
baseline image
new image
visual diff
👤 rejects the difference
Test fails ❌
baseline image
new image
visual diff
👤 accepts the difference
Test passes ✅
baseline image
Accepted image becomes the new baseline image
👩💻 🛑 🕰
🤖 ✅ ⏱
Example: Sudoku with responsive design = 15 images
App at different stages of the game
Tablet resolution
Components
🙁 Hard to test complex app states with dynamic data, clock
😍 — but not with Cypress!
😵 Performance: 100s of images
😡 Consistent & deterministic rendering
😬 Day-to-day review process & approval workflow.
Decision point:
Do-It-Yourself vs 3rd party service
Expensive 💰.
Disclaimer: PhD in Computer Vision and Image Processing
Recommended ✅
it('Loads the homepage', function() {
cy.visit('/')
// Take a snapshot for visual diffing
// https://docs.percy.io/docs/cypress
cy.percySnapshot()
// https://applitools.com/tutorials/cypress.html
cy.eyesCheckWindow()
// https://docs.happo.io/docs/cypress
cy.get('.app').happoScreenshot()
})
cy.screenshot({ capture: 'runner' }) // the test runner
cy.screenshot() // the application
cy.get('.status').screenshot('status') // an element
The Test Runner
cy.screenshot({ capture: 'runner' }) // the test runner
cy.screenshot() // the application
cy.get('.status').screenshot('status') // an element
The Test Runner
The app
cy.screenshot({ capture: 'runner' }) // the test runner
cy.screenshot() // the application
cy.get('.status').screenshot('status') // an element
The Test Runner
The app
An element
yarn add -D cypress-image-snapshot
// cypress/support/index.js
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()
// cypress/plugins/index.js
module.exports = (on, config) => {
require('cypress-react-unit-test/plugins/react-scripts')(on, config)
require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
return config
}
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
it('shows selected number', () => {
mount(
<SudokuContext.Provider value={{ numberSelected: '4' }} >
<div className="innercontainer">
<section className="status">
<Numbers />
</section>
</div>
</SudokuContext.Provider>
)
cy.contains('.status__number', '4')
.should('have.class', 'status__number--selected')
cy.get('.status__numbers')
.matchImageSnapshot('numbers-selected')
})
})
confirm the DOM has been updated
take image snapshot
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
it('shows selected number', () => {
mount(
<SudokuContext.Provider value={{ numberSelected: '4' }} >
<div className="innercontainer">
<section className="status">
<Numbers />
</section>
</div>
</SudokuContext.Provider>
)
cy.contains('.status__number', '4')
.should('have.class', 'status__number--selected')
cy.get('.status__numbers')
.matchImageSnapshot('numbers-selected')
})
})
Commit this image into source control
.status__number {
- padding: 12px 0;
+ padding: 10px 0;
}
App.css
baseline image
new image
difference
cypress/snapshots/Numbers.spec.js/__diff_output__/numbers-selected.diff.png
baseline image
new image
difference
App visual diff due to difference board
Use local Docker container to run Cypress and render consistent images
Mac vs Linux font rendering
Ignore differences < N%
OR
{
"scripts": {
"docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0"
}
}
package.json
"yarn run docker:run"
{
"scripts": {
"docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0"
}
}
package.json
jobs:
cypress-run:
runs-on: ubuntu-latest
container: cypress/browsers:node12.13.0-chrome80-ff74
CI config
exactly the same dependencies
it('saves the page', () => {
// normal Cypress commands
cy.contains('.some-selector', 'some text')
.should('be.visible')
// when the app has reached the desired state
// save the page in the folder "page"
.savePage('page')
})
Do-it-yourself image comparison and management
/// <reference types="cypress" />
describe('Sudoku', () => {
it('shows the initial game', () => {
cy.vrtStart()
// stop the game clock
cy.clock()
// load the initial and solved game boards
cy.fixture('init-array.json').then((initialBoard) => {
cy.fixture('solved-array.json').then((solvedBoard) => {
cy.visit('/', {
onBeforeLoad(win) {
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
})
})
})
// the board fixture has 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
cy.vrtStop()
})
})
Typical End-to-end Visual Test
controls the randomness and the game clock
Cypress Test Runner: first run
VRT dashboard on first run
"docker-compose up"
The screenshot matches the baseline
Run Cypress tests again
onBeforeLoad(win) {
+ initialBoard[2] = 8
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
cy.vrtTrack('sudoku')
if the test or app changes
Image approval workflow
import React from 'react'
import { mockInBundle } from 'mock-in-bundle'
import initialBoard from '../fixtures/init-array.json'
import solvedBoard from '../fixtures/solved-array.json'
describe('Sudoku', () => {
it('mocks Timer and UniqueSudoku', () => {
mockInBundle(
{
'Timer.js': {
Timer: () => <div className="status__time">TIME</div>,
},
'UniqueSudoku.js': {
getUniqueSudoku: () => [initialBoard, solvedBoard],
},
},
'main.chunk.js',
)
cy.visit('/')
cy.contains('.status__time', 'TIME').should('be.visible')
// take visual snapshot
})
})
// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')
By Gleb Bahmutov
Visual testing is extremely effective at confirming the application works and looks the way it did previously, and that the new commits have not accidentally broken it. In this presentation, I will show how to do visual testing using the Cypress test runner, both at the end-to-end and at the individual component levels. We will also consider the trade-offs of doing image diffing ourselves vs paying for a 3rd party service. Presented at ConFoo CA 2022
JavaScript ninja, image processing expert, software quality fanatic