Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end.
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional
(these slides)
50 people. Atlanta, Philly, Boston, NYC, the world
Fast, easy and reliable testing for anything that runs in a browser
$ npm install -D cypress
it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})
$ npx cypress open
Would you sign in?
Can you sign in?
Would you click Buy?
cy.get('.buy')
.should('be.visible')
.and('have.text', 'Buy Now')
cy.get('.secure-note')
.should('be.visible')
.and('have.text', 'Secure transaction')
every element, every style and layout property...
desktop
tablet
mobile
Look at every application flow at every resolution
// import 'todomvc-app-css/index.css'
functional tests still pass!
don't try assert visual look
baseline image
new image
visual diff
baseline image
new image
visual diff
👤 rejects the difference
Test fails ❌
baseline image
new image
visual diff
👤 accepts the difference
Test passes ✅
baseline image
Accepted image becomes the new baseline image
👩💻 🛑 🕰
🤖 ✅ ⏱
Example: Sudoku with responsive design = 15 images
Tablet resolution
Components
🙁 Hard to test complex app states with dynamic data, clock
😍 — but not with Cypress!
😵 Performance: 100s of images
😡 Consistent & deterministic rendering
😬 Day-to-day review process & approval workflow.
Do-It-Yourself vs 3rd party service
Expensive 💰.
Disclaimer: PhD in Computer Vision and Image Processing
Recommended ✅
it('Loads the homepage', function() {
cy.visit('/')
// Take a snapshot for visual diffing
// https://docs.percy.io/docs/cypress
cy.percySnapshot()
// https://applitools.com/tutorials/cypress.html
cy.eyesCheckWindow()
// https://docs.happo.io/docs/cypress
cy.get('.app').happoScreenshot()
})
cy.screenshot({ capture: 'runner' }) // the test runner
cy.screenshot() // the application
cy.get('.status').screenshot('status') // an element
The Test Runner
The app
An element
https://glebbahmutov.com/blog/open-source-visual-testing-of-components/
Mac vs Linux font rendering
https://glebbahmutov.com/blog/open-source-visual-testing-of-components/
Extra complexity 🤯
Do-it-yourself image comparison and management
/// <reference types="cypress" />
describe('Sudoku', () => {
it('shows the initial game', () => {
cy.vrtStart()
// stop the game clock
cy.clock()
// load the initial and solved game boards
cy.fixture('init-array.json').then((initialBoard) => {
cy.fixture('solved-array.json').then((solvedBoard) => {
cy.visit('/', {
onBeforeLoad(win) {
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
})
})
})
// the board fixture has 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
cy.vrtStop()
})
})
Typical End-to-end Visual Test
controls the randomness and the game clock
Cypress Test Runner: first run
VRT dashboard on first run
"docker-compose up"
The screenshot matches the baseline
Run Cypress tests again
onBeforeLoad(win) {
+ initialBoard[2] = 8
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
cy.vrtTrack('sudoku')
if the test or app changes
Image approval workflow
// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')
Image comparison with retries
"How do we know the canvas has finished rendering?"
Source: the blog post https://glebbahmutov.com/blog/canvas-testing/
How does it know when to grab the canvas element to compare?
// this test passes but only because it waits a long time
// for the animation to be finished before taking the image
it('smiles broadly with wait', () => {
cy.visit('/smile')
cy.wait(4000) // ANTI-PATTERN: HARDCODED 4s WAIT
downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
cy.task('compare', { filename })
.should('deep.equal', { match: true })
})
})
If the test waits just 1 second
then the images do not match
cy.get('.todo-list li') // command
.should('have.length', 2) // assertion
import { recurse } from 'cypress-recurse'
it('smiles broadly', () => {
cy.visit('/smile')
// notice: no cy.wait!
recurse(
() => { // Cypress commands to run
return downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
return cy.task('compare', { filename })
})
},
({ match }) => match, // predicate
)
})
https://github.com/bahmutov/cypress-recurse
A way to re-run Cypress commands until a predicate function returns true
import { looksTheSame } from './utils'
it('looks the same', () => {
cy.visit('/smile')
looksTheSame('smile.png')
})
// pixelmatch NPM library runs in the browser
import pixelmatch from 'pixelmatch'
cy.get('canvas').then(($canvas) => {
const ctx1 = $canvas[0].getContext('2d')
const width = $canvas[0].width
const height = $canvas[0].height
const img1 = ctx1.getImageData(0, 0, width, height)
cy.wait(3000)
// the image has finished animation, maybe
cy.get('canvas').then(($canvas) => {
const ctx2 = $canvas[0].getContext('2d')
const img2 = ctx2.getImageData(0, 0, width, height)
const diff = ctx2.createImageData(width, height)
const number = pixelmatch(img1.data, img2.data, diff.data, width, height)
// if number === 0
// images are the same
})
})
recurse(
() => {
return cy.get(selector, noLog).then(($canvas) => {
// got img2
const number = pixelmatch(img1, img2, ...)
// for next comparison, use the new image
// as the base - this way we can get to the end
// of any animation
img1 = img2
return number
})
},
(numberOfDifferentPixels) => numberOfDifferentPixels < 10,
{
// by default uses the default command timeout
log: (numberOfDifferentPixels) =>
cy.log(`**${numberOfDifferentPixels}** diff pixels`),
delay,
},
)
it('can wait for canvas to become static', () => {
cy.visit('/smile')
ensureCanvasStatic('canvas')
// after this it is pretty much guaranteed to
// immediately pass the image diffing on the 1st try
looksTheSame('smile.png', false)
})
it('can wait for canvas to become static', () => {
cy.visit('/smile')
ensureCanvasStatic('canvas')
// after this it is pretty much guaranteed to
// immediately pass the image diffing on the 1st try
looksTheSame('smile.png', false)
})
import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
const log = false
cy.visit('/bar')
looksTheSame('bar-chart.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-added-dataset.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-3-sets.png', log)
// remove first two data sets
cy.contains('button', 'Remove Dataset')
.click().click()
looksTheSame('bar-chart-third-dataset-only.png', log)
})
import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
const log = false
cy.visit('/bar')
looksTheSame('bar-chart.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-added-dataset.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-3-sets.png', log)
// remove first two data sets
cy.contains('button', 'Remove Dataset')
.click().click()
looksTheSame('bar-chart-third-dataset-only.png', log)
})
By Gleb Bahmutov
Visual testing is extremely effective at confirming the application works and looks the way it did previously, and that the new commits have not accidentally broken it. In this presentation, I will show how to do visual testing using Cypress test runner, both at the end-to-end and at the individual component levels. We will also consider the trade-offs of doing image diffing ourselves vs paying for a 3rd party service. Video available at https://youtu.be/KX6Xb5oIaH0
JavaScript ninja, image processing expert, software quality fanatic