Visual Regression Testing
@klamping
http://vrtslides.klamp.in
1
Who I Am
Keywords: Front-end Engineer, Father, Husband, Disc Golfer, Stargazer, Gardener, Gamer, Geek, Soccer Player, Scale Modeller, Piano Player, Spurs Fan, Freelancer, Teacher, Writer, Designer, Developer, RC Plane/Heli Pilot, Writer, et cetera.
@klamping
vrtslides.klamp.in
2
TIFU
@klamping
vrtslides.klamp.in
by fixing the CSS
and breaking production
3
This ticket:
@klamping
vrtslides.klamp.in
"site search is loading ginormous author photos in search results"
4
@klamping
vrtslides.klamp.in
5
@klamping
vrtslides.klamp.in
6
@klamping
vrtslides.klamp.in
7
@klamping
vrtslides.klamp.in
8
@klamping
vrtslides.klamp.in
9
SNAFU
10
Reasons you Don't test
- Ignorance
- Laziness
- All of the above
@klamping
vrtslides.klamp.in
11
Let's make testing less work
@klamping
vrtslides.klamp.in
12
Because I'm lazy okay
@klamping
vrtslides.klamp.in
13
What is visual regression testing?
Image credit WebdriverCSS
@klamping
vrtslides.klamp.in
14
It's visual
Screenshot based (i.e. not unit tests)
Image credit Wraith
@klamping
vrtslides.klamp.in
15
It Tests Regressions
Alerts you when good things go bad
@klamping
vrtslides.klamp.in
16
It's Testing
“Testing shows the presence, not the absence of bugs.”
Edsger W. Dijkstra
@klamping
vrtslides.klamp.in
By Hamilton Richards [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons
17
testing websites is not that easy
[insert cliche "one does not simply" meme]
@klamping
vrtslides.klamp.in
18
websites are all special little snowflakes
@klamping
vrtslides.klamp.in
19
Some Snowflakes
- Dynamic site w/Angular/React/Ember/Backbone/etc
- Simple site that you haven't touched in a year
- 1000+ page CMS
-
Visual/interactive page with animations, background image fanciness, etc
- Parallax/scroll effects
@klamping
vrtslides.klamp.in
20
Some Snowflakes
- Responsive layouts with content that's different per screen resolution even though it's the same page
- Brand new sites under heavy development
- A/B Sites (or multivariate)
- Mobile web apps
- Sites behind logins
- Internal portal with SSO
@klamping
vrtslides.klamp.in
21
Some Snowflakes
- Sites with server/API functionality
- Responses from the server
- Content that loads on scroll
- "Front" pages with content that changes daily
- Twitter or a news page or blog
@klamping
vrtslides.klamp.in
22
Some Snowflakes
- CSS animations with long delays
- Different content between dev and staging and production
- Content that changes automatically on the page (timer/carousel/etc)
@klamping
vrtslides.klamp.in
23
Burn the snowflakes
Let's make a plan
@klamping
vrtslides.klamp.in
24
Start small
@klamping
vrtslides.klamp.in
25
test components,
Not PAGES
@klamping
vrtslides.klamp.in
26
Be selective
priority
complexity
stability
@klamping
vrtslides.klamp.in
27
The tools
- WebdriverCSS
- PhantomCSS
- Gemini
- Wraith
- Applitools Eyes
- DiffUX
- Dpxdt
- Shoov
- Hardy.io
- et cetera
@klamping
vrtslides.klamp.in
28
What we'll do
- Define the site to test
- Set up a local Selenium server
- Define browsers to test in
- Set screenshot areas
- Define responsive breakpoints
- Run the tests
@klamping
vrtslides.klamp.in
29
What we'll do
- Page actions (e.g. mouse clicks)
- Ignore elements on the page
- Hook up to BrowserStack
- Compare and accept changes
@klamping
vrtslides.klamp.in
30
Demo time
@klamping
vrtslides.klamp.in
31
Diving Deeper
Handling changing Content
-
Style-guide based testing
-
Serving Fake data
@klamping
vrtslides.klamp.in
32
Diving Deeper
-
REVIEW PROCESS/TEAM FLOW
-
build system Integration
-
STORING YOUR BASELINE IMAGES
@klamping
vrtslides.klamp.in
33
Final Thoughts
@klamping
vrtslides.klamp.in
34
It's difficult
Keep it fun
- You're teaching a computer how to use your website
- Think about your hook
- Team up
@klamping
vrtslides.klamp.in
35
Take
baby steps
- Play the long game
- Break up the work to form better habits
@klamping
vrtslides.klamp.in
36
It doesn't always work
You won't have perfect tests, but you will have valuable tests
@klamping
vrtslides.klamp.in
37
Learning more
vrtnotes.klamp.in
@klamping
vrtslides.klamp.in
38
Thanks!
@klamping
vrtslides.klamp.in
Visual Regression Testing
By Kevin Lamping
Visual Regression Testing
- 3,545