Gideon Pyzer
Front End Web Developer working in London, with an interest in JavaScript and web technologies.
Visual Regression Testing
by
Gideon Pyzer
Introduction
- Front end developer based in London
- Work at Huddle, building document collaboration application
- Before that, worked at SWG, facilities management
- This is my first talk *gulp*
Blog: https://www.gideonpyzer.com/blog/
Slides: https://tiny.cc/visual-regression-testing
Twitter: @gidztech
Testing
Usually automation tests look at functional reqs of apps
Interesting read: https://martinfowler.com/bliki/UnitTest.html
- End-to-end tests look at whole user flow through the app, not individual components
- Units might be a JS function, class
- Can be a behaviour, feature, UI component
Both can make use of browser automation technology
What's missing?
- You can test if a UI behaves correctly:
- Clicking an item performs an action
- Navigation/routing works as expected
- DOM elements exist on page
- Markup matches a previous snapshot
But how do you test if the application looks right?
Manual regression testing
But what if:
- App is large, loads of different screens, variations
- Support many window sizes/devices, responsive content
- Limited QA
- Manual tester misses something (change blindness, tired)
- Can easily spot some UI regressions
- Might be caught in sanity or smoke tests, or in later in depth test cases
Visual Regression Testing
What if we could take a photo of the app before and after something breaks and compare it in a computery way?
Pixel Comparisons
A pixel consists of a combination of
red, green and blue (RGB)
Simplest form:
- Take each pixel of an image
- Compare it with corresponding pixel in second image
- If any don't match, there's a different in the image
- We can calculate a threshold percentage
Libraries
Anti-aliasing on different environments
can mean different comparison results
- Libraries available which can be configured accordingly:
- Resemble JS
- Pixelmatch
Pixel Comparisons
Spot the difference?
Example from: https://huddleeng.github.io/Resemble.js/
Pixel Comparisons
Example from: https://huddleeng.github.io/Resemble.js/
Visual Regression Testing
Step 1: Take a screenshot of UI when it looks right
Step 2: Version that in Git (or similar) as the baseline image
...
Alternative approach: Compare dev/staging to production, no need for baseline images (full UI)
Step 3: Take a screenshot of the UI again
Step 4: Compare the two images
Step 5: Pass or fail test depending on threshold
Benefits
- It's a form of blackbox testing, only need a DOM selector
- Capture regressions during code/architecture refactoring
- Changes to CSS breaks a component in some modes
- Splitting up CSS into files, diff load order => specificity wars
- Branding testing
- No need to test functional code twice, just how UI looks
- Can be part of your CI/CD flow
Limitations
- Content that changes (on purpose) will break tests, e.g. news
- Animations run at different speeds on different environments
- Screenshots taken at slightly different times may fail test
- Workaround: Disable animations during the test
- Cross browser/environment testing:
- Usually test single browser, expensive to repeat
- Should smoke test other browsers, or use BrowserStack
- UI renders slightly different in Windows/Mac/Linux
PhantomCSS
- Created by James Cryer at Huddle in 2013
- One of the first VRT libraries
- Very popular tool, 4,715 stars on Github
- Built with:
- PhantomJS: a headless browser (no GUI)
- CasperJS: a browser interaction & testing library
- ResembleJS: an image comparison library
- Limitations:
- Modern browser features not supported (CSS Grid)
- PhantomJS is no longer maintained
Muppeteer
- Created by me at Huddle
- Built with:
- Puppeteer: a headless Chrome interaction library
- Mocha/Chai: a test runner and assertion library
- Pixelmatch: an image comparison library
- Built-in Docker support
- Reporting available with JUnit and Mochawesome
- Limitations:
- It's a framework, so not easily pluggable
- No parallelisation, but some optimizations made
- Mostly just maintained by me
Demo
Jest-image-snapshot
- Created by Andres Escobar at American Express
- Built with:
- Jest: a JavaScript testing framework
...
it('renders correctly', async () => {
const page = await browser.newPage();
await page.goto('https://localhost:3000');
const image = await page.screenshot();
expect(image).toMatchImageSnapshot();
});
...
- Limitations:
- Still new, some customisations not available
- Up to you to set up Docker
- Plugin, so easily integrated into Jest workflow
Demo
Libraries
PhantomCSS by Huddle (legacy):
https://github.com/HuddleEng/PhantomCSS
Muppeteer by Huddle
https://github.com/HuddleEng/Muppeteer
Jest-image-snapshot by American Express:
https://github.com/americanexpress/jest-image-snapshot
BackstopJS by Garris:
https://github.com/garris/BackstopJS
More: https://github.com/mojoaxel/awesome-regression-testing
OK, Bye!
Blog: https://www.gideonpyzer.com/blog/
Slides: https://tiny.cc/visual-regression-testing
Twitter: @gidztech
Visual Regression Testing (Halfstack Meetup)
By Gideon Pyzer
Visual Regression Testing (Halfstack Meetup)
Automated testing how the UI looks, not just how it behaves (Rough slides)
