Visual Regression Testing

Gideon Pyzer

Introduction

- Front end developer based in London

- Work at Huddle, building document collaboration application

- Before that, worked at SWG, facilities management

- This is my first talk *gulp*

Blog: https://www.gideonpyzer.com/blog/

Slides: https://tiny.cc/visual-regression-testing

Twitter: @gidztech

Testing

Usually automation tests look at functional reqs of apps

Interesting read: https://martinfowler.com/bliki/UnitTest.html

- End-to-end tests look at whole user flow through the app, not individual components

- Units might be a JS function, class

- Can be a behaviour, feature, UI component

Both can make use of browser automation technology

What's missing?

- You can test if a UI behaves correctly:

- Clicking an item performs an action

- Navigation/routing works as expected

- DOM elements exist on page

- Markup matches a previous snapshot

But how do you test if the application looks right?

Manual regression testing

But what if:

- App is large, loads of different screens, variations

- Support many window sizes/devices, responsive content

- Limited QA

- Manual tester misses something (change blindness, tired)

- Can easily spot some UI regressions

- Might be caught in sanity or smoke tests, or in later in depth test cases

Visual Regression Testing

What if we could take a photo of the app before and after something breaks and compare it in a computery way?

Pixel Comparisons

A pixel consists of a combination of

red, green and blue (RGB)

Simplest form:

- Take each pixel of an image

- Compare it with corresponding pixel in second image

- If any don't match, there's a different in the image

- We can calculate a threshold percentage

Libraries

Anti-aliasing on different environments

can mean different comparison results

- Libraries available which can be configured accordingly:

- Resemble JS

- Pixelmatch

Pixel Comparisons

Spot the difference?

Example from: https://huddleeng.github.io/Resemble.js/

Pixel Comparisons

Example from: https://huddleeng.github.io/Resemble.js/

Visual Regression Testing

Step 1: Take a screenshot of UI when it looks right

Step 2: Version that in Git (or similar) as the baseline image

...

Alternative approach: Compare dev/staging to production, no need for baseline images (full UI)

Step 3: Take a screenshot of the UI again

Step 4: Compare the two images

Step 5: Pass or fail test depending on threshold

Benefits

- It's a form of blackbox testing, only need a DOM selector

- Capture regressions during code/architecture refactoring

- Changes to CSS breaks a component in some modes

- Splitting up CSS into files, diff load order => specificity wars

- Branding testing

- No need to test functional code twice, just how UI looks

- Can be part of your CI/CD flow

Limitations

- Content that changes (on purpose) will break tests, e.g. news

- Animations run at different speeds on different environments

- Screenshots taken at slightly different times may fail test

- Workaround: Disable animations during the test

- Cross browser/environment testing:

- Usually test single browser, expensive to repeat

- Should smoke test other browsers, or use BrowserStack

- UI renders slightly different in Windows/Mac/Linux

PhantomCSS

- Created by James Cryer at Huddle in 2013

- One of the first VRT libraries

- Very popular tool, 4,715 stars on Github

- Built with:

- PhantomJS: a headless browser (no GUI)

- CasperJS: a browser interaction & testing library

- ResembleJS: an image comparison library

- Limitations:

- Modern browser features not supported (CSS Grid)

- PhantomJS is no longer maintained

Muppeteer

- Created by me at Huddle

- Built with:

- Puppeteer: a headless Chrome interaction library

- Mocha/Chai: a test runner and assertion library

- Pixelmatch: an image comparison library

- Built-in Docker support

- Reporting available with JUnit and Mochawesome

- Limitations:

- It's a framework, so not easily pluggable

- No parallelisation, but some optimizations made

- Mostly just maintained by me

Demo

Example:

https://github.com/gidztech/vrt-examples/tree/master/examples/muppeteer-example

Jest-image-snapshot

- Created by Andres Escobar at American Express

- Built with:

- Jest: a JavaScript testing framework

  ...
  it('renders correctly', async () => {
    const page = await browser.newPage();
    await page.goto('https://localhost:3000');
    const image = await page.screenshot();

    expect(image).toMatchImageSnapshot();
  });
  ...

- Limitations:

- Still new, some customisations not available

- Up to you to set up Docker

- Plugin, so easily integrated into Jest workflow

Demo

Example:

https://github.com/gidztech/vrt-examples/tree/master/examples/jest-image-snapshot-example

Libraries

PhantomCSS by Huddle (legacy):

https://github.com/HuddleEng/PhantomCSS

Muppeteer by Huddle

https://github.com/HuddleEng/Muppeteer

Jest-image-snapshot by American Express:

https://github.com/americanexpress/jest-image-snapshot

BackstopJS by Garris:

https://github.com/garris/BackstopJS

More: https://github.com/mojoaxel/awesome-regression-testing

OK, Bye!

Blog: https://www.gideonpyzer.com/blog/

Slides: https://tiny.cc/visual-regression-testing

Twitter: @gidztech