Visual Regression Testing

Gideon Pyzer

@gidztech

Introduction

- Front end web developer based in London

- Work at , building document collaboration application

- I'm new to public speaking

@gidztech

Gideon Pyzer

UI Testing

You can test if the UI behaves correctly:

@gidztech

- Components react to routing changes

- DOM elements exist on page

- Clicking a button performs an action

But how do you test if the application looks right?

Manual testing

You might spot colour changes?

@gidztech

What if...

@gidztech

- App is large, many different screens

- Many window sizes/devices, responsive

- Limited QA time

- Miss something, change blindness

What if we could take a photo of the app before and after something breaks...

@gidztech

... and compare it in a computery way?

@gidztech

Pixel Comparisons

@gidztech

A pixel consists of a combination of red, green and blue (RGB)

Algorithm: Take pixel in image one, compare RGB value at corresponding location in image two

The Process

@gidztech

Step 1: Take a screenshot of UI when it looks right

Step 2: Version that in Git (or similar) as the baseline image

Oh no, something broke!

Step 3: Take a screenshot of the UI again

The Process (continued)

@gidztech

Step 4: Compare the two images

Step 5: Pass or fail test depending on threshold set (e.g. 1%)

- A form of testing, only need a DOM selector

Benefits

@gidztech

- Capture regressions during code/architecture refactoring

- Changing CSS breaks a component in some cases

- Test theming/branding

- Splitting up CSS into files, diff load order...

SPECIFICITY WARS!

- Content that changes (on purpose) will break tests, e.g. news

Limitations

@gidztech

- Animations run at different speeds

- Screenshots taken at slightly different times may fail test

Workaround: Disable animations during the test

- Cross environment testing

- UI renders slightly different in Windows/Mac/Linux

Created by James Cryer at

@gidztech

Problems

- Modern browser features not supported (CSS Grid)

- PhantomJS is no longer maintained

PhantomJS

CasperJS

ResembleJS

Created by me

@gidztech

Puppeteer

Mocha/Chai

Pixelmatch

Docker

Example

https://github.com/gidztech/vrt-examples/tree/master/examples/muppeteer-example

Example Component

@gidztech

const Panel = props => {
    return (
        <div className="panel">
            <div className="panel-heading">
                { props.icon && (
                    <span className="panel-icon">
                        <ion-icon name={props.icon} />
                    </span>
                ) }
                <h1 className="panel-title">My title</h1>
            </div>
            <div className="panel-body">{props.children}</div>
        </div>
    );
};

Example Component

@gidztech

Simple mode

Fancy icon mode

Example Tests

@gidztech

// Test 1
describe('Simple mode', async () => {
    it('title exists', async () => {
        const titleText = await page.extensions.getText(panelTitle);
        assert.equal(titleText, 'My title');
    });
    it('title appears correctly', async () => {
        await assert.visual(panelContainer);
    });
});

// Test 2
describe('Icon mode', async () => {
    it('title exists', async () => {
        const titleText = await page.extensions.getText(panelTitle);
        assert.equal(titleText, 'My title');
    });
    it('title and icon appear correctly', async () => {
        await assert.visual(panelContainer);
    });
});