Visual Regression Testing

https://github.com/visualtesting

Eli White

Why Visual Regression Testing?

Refactoring CSS is scary

 

Consistently test changes in browsers most devs don't use

 

Make sure that changing a component in a style guide doesn't have unintended consequences

 

Manual testing sucks.

Importance Hierarchy

CSS / SASSlint → mock browser → selenium → manual (visual)  visual (automatic) → visual (manual)

Quality Screenshots

Multiple browsers (Chrome, IE, Safari, Firefox)

 

Multiple resolutions (css breakpoints)

Testing a full page

Testing a single unit

Question...

What about dynamic sections? 

3 options.

Quality Image Diffs

ImageMagick is the common solution (slow, manual installation)

 

Done as a service: Applitools

 

Done in a browser: resemble.js (canvas)

 

Done in node: node-resemble

...uses a canvas implementation written with ImageMagick

 

Pure javascript implementation

Reviewing

If reviewing diffs is optional, it won't be done

 

Require manual review, but low bar to accept

...

"Approve Button"

Admin Panel

Engineering Goals

  • Take screenshots in different languages (js / ruby / obj-c)
  • Different Source Repository hosts (Github / Stash / Bitbucket)
  • Store images in different places (fs, amazon, git)
  • Diff against specified builds
  • Different tools to generate the diff images

Plugins all the way down

Plugins all the way down

  • visualtesting-storage-fs
  • visualtesting-storage-s3
  • visualtesting-storage-git
  • visualtesting-diff-imagemagick
  • visualtesting-diff-resemble
  • visualtesting-diff-applitools
  • visualtesting-service-github
  • visualtesting-service-stash
  • visualtesting-service-bitbucket

What keeps me up at night?

Speed: n-pages * m-browsers * k-builds 

Luckily there are lots of ways to parallelize this

A name. Seriously. VisualDiffing is *not* sexy

Noise / False positives 

Luckily there are lots of low effort ways to improve the diffing option

What's next?

Some code cleanup

 

Lets start using it on some ideal projects (roswell)

 

Documentation

 

Try using it with native apps

 

 

Thanks

Special thanks to Christian Bromann

http://github.com/VisualTesting

<-- Seriously. I need a new name and logo. I stole this off Google Images

VisualTesting

By Eli White

VisualTesting

  • 401