Visual Regression Testing

@klamping

http://vrtslides.klamp.in

1

Who I Am

Keywords: Front-end Engineer, Father, Husband, Disc Golfer, Stargazer, Gardener, Gamer, Geek, Soccer Player, Scale Modeller, Piano Player, Spurs Fan, Freelancer, Teacher, Writer, Designer, Developer, RC Plane/Heli Pilot, Writer, ​et cetera.​

@klamping

vrtslides.klamp.in

2

TIFU

@klamping

vrtslides.klamp.in

by fixing the CSS

and breaking production

3

This ticket:

@klamping

vrtslides.klamp.in

"site search is loading ginormous author photos in search results"

4

@klamping

vrtslides.klamp.in

5

@klamping

vrtslides.klamp.in

6

@klamping

vrtslides.klamp.in

7

@klamping

vrtslides.klamp.in

8

@klamping

vrtslides.klamp.in

9

SNAFU

10

Reasons you Don't test

  • Ignorance
  • Laziness
  • All of the above

@klamping

vrtslides.klamp.in

11

Let's make testing less work

@klamping

vrtslides.klamp.in

12

Because I'm lazy okay

@klamping

vrtslides.klamp.in

13

What is visual regression testing?

Image credit WebdriverCSS

@klamping

vrtslides.klamp.in

14

It's visual

Screenshot based (i.e. not unit tests)

Image credit Wraith

@klamping

vrtslides.klamp.in

15

It Tests Regressions

Alerts you when good things go bad

@klamping

vrtslides.klamp.in

16

It's Testing

“Testing shows the presence, not the absence of bugs.”

Edsger W. Dijkstra

@klamping

vrtslides.klamp.in

By Hamilton Richards [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons

17

testing websites is not that easy

[insert cliche "one does not simply" meme]

@klamping

vrtslides.klamp.in

18

websites are all special little snowflakes

@klamping

vrtslides.klamp.in

19

Some Snowflakes

  • Dynamic site w/Angular/React/Ember/Backbone/etc
  • Simple site that you haven't touched in a year
  • 1000+ page CMS
  • Visual/interactive page with animations, background image fanciness, etc
    • Parallax/scroll effects

@klamping

vrtslides.klamp.in

20

Some Snowflakes

  • Responsive layouts with content that's different per screen resolution even though it's the same page
  • Brand new sites under heavy development
  • A/B Sites (or multivariate)
  • Mobile web apps
  • Sites behind logins
    • Internal portal with SSO

@klamping

vrtslides.klamp.in

21

Some Snowflakes

  • Sites with server/API functionality
    • Responses from the server
  • Content that loads on scroll
  • "Front" pages with content that changes daily
    • Twitter or a news page or blog

@klamping

vrtslides.klamp.in

22

Some Snowflakes

  • CSS animations with long delays
  • Different content between dev and staging and production
  • Content that changes automatically on the page (timer/carousel/etc)

@klamping

vrtslides.klamp.in

23

Burn the snowflakes

Let's make a plan

@klamping

vrtslides.klamp.in

24

Start small

@klamping

vrtslides.klamp.in

25

test components,

Not PAGES

@klamping

vrtslides.klamp.in

26

Be selective

priority

complexity

stability

@klamping

vrtslides.klamp.in

27

The tools

  • WebdriverCSS
  • PhantomCSS
  • Gemini
  • Wraith
  • Applitools Eyes
  • DiffUX
  • Dpxdt
  • Shoov
  • Hardy.io
  • et cetera

@klamping

vrtslides.klamp.in

28

What we'll do

  • Define the site to test
  • Set up a local Selenium server
  • Define browsers to test in
  • Set screenshot areas
  • Define responsive breakpoints
  • Run the tests

@klamping

vrtslides.klamp.in

29

What we'll do

  • Page actions (e.g. mouse clicks)
  • Ignore elements on the page
  • Hook up to BrowserStack
  • Compare and accept changes

@klamping

vrtslides.klamp.in

30

Demo time

@klamping

vrtslides.klamp.in

31

Diving Deeper

Handling changing Content

  • Style-guide based testing

  • Serving Fake data

@klamping

vrtslides.klamp.in

32

Diving Deeper

  • REVIEW PROCESS/TEAM FLOW 

  • build system Integration

  • STORING YOUR BASELINE IMAGES

@klamping

vrtslides.klamp.in

33

Final Thoughts

@klamping

vrtslides.klamp.in

34

It's difficult
Keep it fun

  • You're teaching a computer how to use your website
  • Think about your hook
  • Team up

@klamping

vrtslides.klamp.in

35

Take
baby steps

  • Play the long game
  • Break up the work to form better habits

@klamping

vrtslides.klamp.in

36

It doesn't always work

You won't have perfect tests, but you will have valuable tests

@klamping

vrtslides.klamp.in

37

Learning more

vrtnotes.klamp.in

@klamping

vrtslides.klamp.in

38

Thanks!

@klamping

vrtslides.klamp.in

Visual Regression Testing

By Kevin Lamping

Visual Regression Testing

  • 3,101
Loading comments...

More from Kevin Lamping