https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end.
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional
(these slides)
50 people. Atlanta, Philly, Boston, NYC, the world
Fast, easy and reliable testing for anything that runs in a browser
$ npm install -D cypress
it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})
$ npx cypress open
Would you sign in?
Can you sign in?
Would you click Buy?
cy.get('.buy')
.should('be.visible')
.and('have.text', 'Buy Now')
cy.get('.secure-note')
.should('be.visible')
.and('have.text', 'Secure transaction')
every element, every style and layout property...
desktop
tablet
mobile
Look at every application flow at every resolution
// import 'todomvc-app-css/index.css'
functional tests still pass!
don't try assert visual look
baseline image
new image
visual diff
baseline image
new image
visual diff
👤 rejects the difference
Test fails ❌
baseline image
new image
visual diff
👤 accepts the difference
Test passes ✅
baseline image
Accepted image becomes the new baseline image
👩💻 🛑 🕰
🤖 ✅ ⏱
Example: Sudoku with responsive design = 15 images
Tablet resolution
Components
🙁 Hard to test complex app states with dynamic data, clock
😍 — but not with Cypress!
😵 Performance: 100s of images
😡 Consistent & deterministic rendering
😬 Day-to-day review process & approval workflow.
Do-It-Yourself vs 3rd party service
Expensive 💰.
Disclaimer: PhD in Computer Vision and Image Processing
Recommended ✅
it('Loads the homepage', function() {
cy.visit('/')
// Take a snapshot for visual diffing
// https://docs.percy.io/docs/cypress
cy.percySnapshot()
// https://applitools.com/tutorials/cypress.html
cy.eyesCheckWindow()
// https://docs.happo.io/docs/cypress
cy.get('.app').happoScreenshot()
})
cy.screenshot({ capture: 'runner' }) // the test runner
cy.screenshot() // the application
cy.get('.status').screenshot('status') // an element
The Test Runner
The app
An element
https://glebbahmutov.com/blog/open-source-visual-testing-of-components/
Mac vs Linux font rendering
https://glebbahmutov.com/blog/open-source-visual-testing-of-components/
Extra complexity 🤯
Do-it-yourself image comparison and management
/// <reference types="cypress" />
describe('Sudoku', () => {
it('shows the initial game', () => {
cy.vrtStart()
// stop the game clock
cy.clock()
// load the initial and solved game boards
cy.fixture('init-array.json').then((initialBoard) => {
cy.fixture('solved-array.json').then((solvedBoard) => {
cy.visit('/', {
onBeforeLoad(win) {
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
})
})
})
// the board fixture has 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
cy.vrtTrack('sudoku') // wrapper around cy.screenshot()
cy.vrtStop()
})
})
Typical End-to-end Visual Test
controls the randomness and the game clock
Cypress Test Runner: first run
VRT dashboard on first run
"docker-compose up"
The screenshot matches the baseline
Run Cypress tests again
onBeforeLoad(win) {
+ initialBoard[2] = 8
win.__initArray = initialBoard
win.__solvedArray = solvedBoard
},
+ cy.get('.status__difficulty-select').select('Medium')
+ cy.get('.status__action-mistakes-mode-switch').click()
cy.vrtTrack('sudoku')
if the test or app changes
Image approval workflow
// functional assertions
// to make sure the app has rendered
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00').should('be.visible')
cy.contains('.status__difficulty-select', 'Easy').should('be.visible')
// followed by the visual assertion
cy.vrtTrack('sudoku')
Image comparison with retries
"How do we know the canvas has finished rendering?"
Source: the blog post https://glebbahmutov.com/blog/canvas-testing/
How does it know when to grab the canvas element to compare?
// this test passes but only because it waits a long time
// for the animation to be finished before taking the image
it('smiles broadly with wait', () => {
cy.visit('/smile')
cy.wait(4000) // ANTI-PATTERN: HARDCODED 4s WAIT
downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
cy.task('compare', { filename })
.should('deep.equal', { match: true })
})
})
If the test waits just 1 second
then the images do not match
cy.get('.todo-list li') // command
.should('have.length', 2) // assertion
import { recurse } from 'cypress-recurse'
it('smiles broadly', () => {
cy.visit('/smile')
// notice: no cy.wait!
recurse(
() => { // Cypress commands to run
return downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
return cy.task('compare', { filename })
})
},
({ match }) => match, // predicate
)
})
https://github.com/bahmutov/cypress-recurse
A way to re-run Cypress commands until a predicate function returns true
import { looksTheSame } from './utils'
it('looks the same', () => {
cy.visit('/smile')
looksTheSame('smile.png')
})
// pixelmatch NPM library runs in the browser
import pixelmatch from 'pixelmatch'
cy.get('canvas').then(($canvas) => {
const ctx1 = $canvas[0].getContext('2d')
const width = $canvas[0].width
const height = $canvas[0].height
const img1 = ctx1.getImageData(0, 0, width, height)
cy.wait(3000)
// the image has finished animation, maybe
cy.get('canvas').then(($canvas) => {
const ctx2 = $canvas[0].getContext('2d')
const img2 = ctx2.getImageData(0, 0, width, height)
const diff = ctx2.createImageData(width, height)
const number = pixelmatch(img1.data, img2.data, diff.data, width, height)
// if number === 0
// images are the same
})
})
recurse(
() => {
return cy.get(selector, noLog).then(($canvas) => {
// got img2
const number = pixelmatch(img1, img2, ...)
// for next comparison, use the new image
// as the base - this way we can get to the end
// of any animation
img1 = img2
return number
})
},
(numberOfDifferentPixels) => numberOfDifferentPixels < 10,
{
// by default uses the default command timeout
log: (numberOfDifferentPixels) =>
cy.log(`**${numberOfDifferentPixels}** diff pixels`),
delay,
},
)
it('can wait for canvas to become static', () => {
cy.visit('/smile')
ensureCanvasStatic('canvas')
// after this it is pretty much guaranteed to
// immediately pass the image diffing on the 1st try
looksTheSame('smile.png', false)
})
it('can wait for canvas to become static', () => {
cy.visit('/smile')
ensureCanvasStatic('canvas')
// after this it is pretty much guaranteed to
// immediately pass the image diffing on the 1st try
looksTheSame('smile.png', false)
})
import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
const log = false
cy.visit('/bar')
looksTheSame('bar-chart.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-added-dataset.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-3-sets.png', log)
// remove first two data sets
cy.contains('button', 'Remove Dataset')
.click().click()
looksTheSame('bar-chart-third-dataset-only.png', log)
})
import { looksTheSame } from './utils'
// a single test can perform multiple comparisons
it('adds and removes data sets', () => {
const log = false
cy.visit('/bar')
looksTheSame('bar-chart.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-added-dataset.png', log)
cy.contains('button', 'Add Dataset').click()
looksTheSame('bar-chart-3-sets.png', log)
// remove first two data sets
cy.contains('button', 'Remove Dataset')
.click().click()
looksTheSame('bar-chart-third-dataset-only.png', log)
})