Visual Regression Testing

https://github.com/visualtesting

Eli White

Why Visual Regression Testing?

Refactoring CSS is scary

Consistently test changes in browsers most devs don't use

Make sure that changing a component in a style guide doesn't have unintended consequences

Manual testing sucks.

Importance Hierarchy

CSS / SASSlint → mock browser → selenium → ~~manual (visual)~~ visual (automatic) → visual (manual)

Quality Screenshots

Multiple browsers (Chrome, IE, Safari, Firefox)

Multiple resolutions (css breakpoints)

Testing a full page

Testing a single unit

Question...

What about dynamic sections?

3 options.

Quality Image Diffs

ImageMagick is the common solution (slow, manual installation)

Done as a service: Applitools

Done in a browser: resemble.js (canvas)

Done in node: node-resemble

...uses a canvas implementation written with ImageMagick

Pure javascript implementation

Reviewing

If reviewing diffs is optional, it won't be done

Require manual review, but low bar to accept

...

"Approve Button"

Admin Panel

Engineering Goals

Take screenshots in different languages (js / ruby / obj-c)
Different Source Repository hosts (Github / Stash / Bitbucket)
Store images in different places (fs, amazon, git)
Diff against specified builds
Different tools to generate the diff images

Plugins all the way down

visualtesting-storage-fs
visualtesting-storage-s3
visualtesting-storage-git
visualtesting-diff-imagemagick
visualtesting-diff-resemble
visualtesting-diff-applitools
visualtesting-service-github
visualtesting-service-stash
visualtesting-service-bitbucket

What keeps me up at night?

Speed: n-pages * m-browsers * k-builds

Luckily there are lots of ways to parallelize this

A name. Seriously. VisualDiffing is *not* sexy

Noise / False positives

Luckily there are lots of low effort ways to improve the diffing option

What's next?

Some code cleanup

Lets start using it on some ideal projects (roswell)

Documentation

Try using it with native apps

Thanks

Special thanks to Christian Bromann

http://github.com/VisualTesting

<-- Seriously. I need a new name and logo. I stole this off Google Images