Visual Testing

with cypress.io

Przemek Suchodolski
@przemuh

About me

  • @przemuh / przemuh.dev
  • Sénior Software Engineer @Egnyte
  • Frontend dev since 2012

  • 🇰🇷 korpo, 🇵🇱 software-house, 🇺🇸 startup/korpo

  • Musician amateur 🎸 🥁 🎹 (soundcloud.com/przemuh)

Visual testing

Visual regression testing

Automated UI testing

it("should have a proper className", () => {
    const myComponent = render();
    
    expect(myComponent).toHaveClass("pink-blink");
})
.parent {
    .pink-blink {
        color: red;
        animation: none;
    }
}

Code review

Refactoring CSS

Visual Testing

basics

Snapshots 📸

Visual tests

  • don't fail
  • don't pass
  • human component is crucial

Pros

  • Catching visual bugs
  • Time save during manual QA
  • Confidence when refactoring
  • UI/UX consistency/review
  • Writing fewer functional tests

Challenges

  • Where to store snapshots?
  • How to review the results of the tests?
  • What about false-positives/false-negatives?
  • Smart testing? AI? Grouping similar changes?

Visual testing in

describe("Cypress test suite", () => {

  it("test case", () => {
  
    cy.server();
    cy.route("/api/some/REST/endpoint", { response: [1,2,3] });
        
    cy.visit("#/my-page");
        
    cy.findByText("Something").should("be.visible");
        
  });
});

Setup

  1. npm install
  2. plugins/index.js
  3. support/commands.js
// home-page.spec.js
describe("Home page", () => {
  
  it("for logged-in user", () => {
    cy.login("admin")
    cy.vistit("/home")
    
    cy.matchImageSnapshot() // "Home page for logged-in user.png"

    cy.matchImageSnapshot(name);
    cy.matchImageSnapshot(options);
    cy.matchImageSnapshot(name, options);
  })
})

Baseline

Current snapshot

Diff

Results

iTerm

Success

🤔

  • When to generate snapshots?
  • How to review/approve them?
  • Where to store them?
  • What about local dev?
  • Merge request?
  • Prepare gallery?
  • git-lfs?
  • ?

v.X

v.Y

WHAT!?

🔎 Searching

for cloud service

Requirements

  • Quick setup ⚡️
  • Relatively cheap 💰
  • Solid 🚀

awesome-*

awesome-regression-testing

⚡️ quick setup - 😢

💰cheap - 😢

🚀 solid - 😢

~21 online services

  • Free plan
  • 10  unlimited users
  • 5000 snapshots
  • 29$ / mo - 10k snapshots
  • Cypress integration
  • Puppeteer, WebdriverIO, Nightmare, Nightwatch, Protractor, TestCafe, Storybook

How does percy work?

percy agent

cy.percySnapshot();

cypress runner

Visual rendering & testing infrastructure

percy builds

DOM + assets (SHA256)

Pros

  • Snapshot grouping
  • Smart baseline picking
  • Automatic animation freezing *css
  • Testing across different device sizes
  • Ignoring 1px diffs *within 7px surrounding circle
  • Builds expire in 30 days / 1 year (depends on plan)

Setup

  1. npm install
  2. plugins/index.js
  3. support/commands.js
  4. cypress run -> percy exec -- cypress run
// home-page.spec.js
describe("Home page", () => {
  
  it("for logged-in user", () => {
    cy.login("admin")
    cy.vistit("/home")
    
    cy.percySnapshot() // "Home page for logged-in user.png"
    cy.percySnapshot(name);
    cy.percySnapshot(options);
    
    cy.percySnapshot("Homepage responsive test", { widths: [768, 992, 1200] });
  })
})

Can I have both? 🤔

local comparison

cloud comparison

Cypress.Commands.add("vrt", (...args) => {
    let name;
    let options;

    if (typeof args[0] === "object") {
        options = args[0];
    } else {
        [name, options] = args;
    }

    // Set the proper viewport height for local vrt when minHeight-percy option is set
    if (options && options.minHeight) {
        const PERCY_MAX_HEIGHT = 2000;
        if (options.minHeight > PERCY_MAX_HEIGHT) {
            throw new Error(
                `Your minHeight ${options.minHeight} is greater than 
				PERCY_MAX_HEIGHT=${PERCY_MAX_HEIGHT}`,
            );
        }

        cy.viewport(Cypress.config("viewportWidth"), options.minHeight);
    }

    if (Cypress.env("percy")) {
        cy.percySnapshot(name, options);
    } else {
        cy.matchImageSnapshot(name, options);
    }
});
percy exec -- cypress run --env percy=true

VRT flow

👩‍💻

generate baseline
snapshots

local comparison

Merge Request

label: vrt

🏁

VRT flow

6am

branch: develop

255 snapshots

~ 15min (including app build)

Watch out for it!

aka - good practicies

cypress open

(window size)

Disable none-css animations

import Highcharts from "highcharts";

Highcharts.setOptions({
    chart: {
        animation: false,
    },
    plotOptions: {
        series: {
            animation: false,
        },
    },
});

//webpack.config

{
  ...,
  entry: [
    "core-js/stable",
    "regenerator-runtime/runtime",
    "custom-event-polyfill",
    "isomorphic-fetch",
    
    target === "test" && "./src/main/javascript/uiTestEntry.js",
    
    "./src/main/javascript/main.js",
  ].filter(Boolean),
}

Data based on date

cy.clock

// setTimeout, setInterval etc. ...
cy.clock(Date.UTC(2020, 1, 11)); 

// only Date
cy.clock(Date.UTC(2020, 1, 11), ["Date"]);

Stop the time with cy.clock (przemuh.dev)

Waiting for DOM element

cy.route("/api/something").as("APICall");

cy.wait("@APICall");

cy.doSnapshot();
cy.route("/api/something").as("APICall");

cy.findByText("Text to wait for");
// or
cy.getByTestId("spinner").should("not.be.visible");

cy.doSnapshot();

Scroll position

cy.get('footer').scrollIntoView()

Where to start?

  • Are most important to revenue and user experience
  • Have the most teams working on them
  • Are the most brittle or have the most legacy code
  • Have the lowest test coverage

Pages that:

Thank you!

Visual Testing with cypress.io

By Przemek Suchodolski

Made with Slides.com

Visual Testing with cypress.io

  • 778

More from Przemek Suchodolski