Phantom Flow

UI testing framework based on decision trees

"A NodeJS wrapper for PhantomJS, CasperJS and PhantomCSS, PhantomFlow enables a fluent way of describing user flows in code whilst generating structured tree data for visualisation."

 

 

 

 

 

 

 

 

https://github.com/Huddle/PhantomFlow

"A NodeJS wrapper for PhantomJS, CasperJS and PhantomCSS, PhantomFlow enables a fluent way of describing user flows in code whilst generating structured tree data for visualisation."

 

 

 

 

 

 

 

 

https://github.com/Huddle/PhantomFlow

Source: https://ifunny.co/fun/hUDAM5X74

Phantom JS

- Headless browser

    - Browser runs without a GUI

- Based on WebKit

   - The rendering engine used by Safari, and formerly Chrome

- Provides JavaScript API for interacting with the engine and page

Source: http://blog.creativeoutlet.com/headless-cartoon-characters/

var page = require('webpage').create();
page.open('https://www.huddle.com/', function (status) {
    if (status !== 'success') {
        console.log('Unable to access network');
    } else {
        var headerText = page.evaluate(function () {
            var el = document.querySelector('.hero-text h2');
            return el.textContent;
        });
        console.log(headerText);
    }
    phantom.exit();
});

Example

Casper JS

- Extends the API provided by Phantom JS

    - Provides a richer set of interaction functions with the page

- Provides a testing library

var casper = require('casper').create();
casper.start('https://www.huddle.com/', function () {
    this.waitUntilVisible('.hero-text h2', function () {
        this.test.pass('Header text is visible');
    }, function () {
        this.test.fail('Header text is not visible');
    }, 2000);
});

Casper JS API

waitForSelector

waitUntilVisible

waitWhileVisible
click

sendKeys
...

 

assert
assertEquals

pass
fail 

Docs: http://docs.casperjs.org/en/latest/modules/index.html

Testing

- Phantom JS - The headless browser

- Casper JS - The interaction and testing library on top

 

We have the tools to write UI tests.

That's it, right?

 

Most developers/testers using this tech stack might agree

 

But we're better than that...

Phantom Flow

- Phantom Flow is a testing framework that allows you to describe user flows, using Decision Trees.

 

 

 

- Tests are usually written in flat, list-like structure

    - But humans think/remember better with trees

 

 

- Benefits

    - Less likely to miss scenarios

    - Good way to learn the code base

    - Can serve as documentation

Terminology (API)

Flow - The name of the test suite, typically a single component

 

Step - A single action in the user flow

 

Decision - A decision point for the user; do action A => some steps; do action B => other steps

 

Chance - Like a decision but used to describe factors outside the control of the user, e.g. error on the page vs no error on page

Example

flow("Get a coffee", function(){
	step("Go to the kitchen", goToKitchen);
	step("Go to the coffee machine", goToMachine);
	decision({
		"Wants Latte": function(){
			chance({
				"There is no milk": function(){
					step("Request Latte", requestLatte_fail);
					decision({
						"Give up": function(){
							step("Walk away from the coffee machine", walkAway);
						},
						"Wants Espresso instead": wantsEspresso
					});
				},
				"There is milk": function(){
					step("Request Latte", requestLatte_success);
				}
			});
		},
		"Wants Cappuccino": function(){
			chance({
				"There is no milk": function(){
					step("Request Cappuccino", requestCappuccino_fail);
					decision({
						"Request Espresso instead": wantsEspresso
					});
				},
				"There is milk": function(){
					step("Request Cappuccino", requestCappuccino_success);
				}
			});
		},
		"Wants Espresso": wantsEspresso
	});
});

Assertions

function goToMachine (){

    casper.click('#coffeemachinebutton');
    casper.waitForSelector(
        '#myModal:not([style*="display: none"])', 
        function success(){
            phantomCSS.screenshot('#myModal .modal-content');
            casper.test.pass('Should see coffee machine');
        },
        function timeout(){
            casper.test.fail('Should see coffee machine');
        }
    );
}

 

casper.test.pass() is an explicit pass for the test, in cases where you don't run standard asserts, such as casper.test.assertEquals()

Assertions

function goToMachine (){

    casper.click('#coffeemachinebutton');
    casper.waitForSelector(
        '#myModal:not([style*="display: none"])', 
        function success(){
            phantomCSS.screenshot('#myModal .modal-content');
            casper.test.pass('Should see coffee machine');
        },
        function timeout(){
            casper.test.fail('Should see coffee machine');
        }
    );
}

 

What is this crazy Phantom CSS thingy?

Visual Regression

- Unit tests will cover functional code

- We can test whether a value is as expected

- We can test that an element exists on the page

 

But how do we verify that it looks right?

 

- We can't easily test that CSS classes and style rules are being applied correctly under the current testing model.

 

The answer is Phantom CSS! 

 

Phantom CSS

casper.then('https://www.huddle.com/', function () {
    phantomCSS.screenshot(selector);

});

- The first time the screenshot function is invoked, the baseline is created. This should be versioned in GIT.

- Subsequent times, a comparison is made (using Resemble.js)

- If the RGB pixel difference is significant enough, the test will fail

- Phantom CSS is a library that compares a baseline screenshot of a component with a new screenshot taken at the time of the test

Phantom CSS

- The first image is the baseline (correct) visual

- The second image is the latest screenshot taken during a test

- The third image shows the differences, outlined in pink 

There are position or padding issues indicated here

If you don't like pink, no problem, this be customised :)

Phantom CSS

What if we change the component?

 

- Failing visual tests don't always mean the component is broken

    - We sometimes re-design or move things about

- Solution: Delete the baseline visual

    - A new one is generated for versioning next time the test is run

Phantom CSS

Considerations

- Visual tests are not as fast as functional JS tests

    - Only use them where necessary, not every element!

- A full page screenshot may not pick up some changes

    - The threshold is a percentage of the number of pixel diffs

    - Better to be taken at the component level (e.g. widgets)

Dev/Tests Huddle

Torque

- Our component library, made up of widgets and components

    - Confused?

    - Huddle Terminology:

        - Component - UI element(s) with no APIs/data ("dumb")

        - Widget - UI element(s), with APIs and data, composed

- Develop our components and widgets in isolation

    - Mock APIs and data

- Use our develop environment (page fixtures, mocking) for tests

    - Easy to re-produce problems found in tests

Component Dev

- A component will typically have:

    - HTML file - the markup

    - Knockout JS file - the JavaScript and component binding

    - SASS file - CSS with variables, mixins, and other goodies

    - YAML config file - Describes inputs, states, implementations

Component Dev

- Describe the inputs for the component

- Describe the implementation, currently always Knockout

Component Dev

- Components are built dynamically using the YAML config file

- Run:                        to get hot docs (webpack dev server)

    - Documentation of our components and all the variations

grunt dev:docs

Component Tests

- Running the tests

 

- Running the report

 

 

 

 

 

- See the baseline image, the new image, and the fail image

- Use Rebase button to replace the baseline image with current

  

grunt test
grunt test --report=true

Widget Dev

- A widget will typically have:

    - HTML file - the markup

    - Knockout JS file - the JavaScript and component binding

    - SASS file - CSS with variables, mixins, and other goodies

    - Develop file - API/data mocks

    - Tests folder - with a test file + compiled version

Source: http://blog.interestship.com

Example: Folder Picker

if (!window.undertest) {
    setup.run('Happy path');
}

setup('Happy path', () => {
    get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi(defaultFolderMock));
    get(/\/files\/pagedfolders\/\d+/, function*() {
        while(true) {
            yield { status:200, response:folderApi(fleRequestsFolderMock) };
        }
    });
}, 'init', 'en-gb');

setup('Pick File requests folder, then go back to root', () => {
    get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi(defaultFolderMock));
    get(/\/files\/pagedfolders\/\d+/, function*() {
        yield {status:200,response:folderApi(fleRequestsFolderMock)};

        while(true) {
            yield { status:200, response:folderApi(defaultFolderMock) };
        }
    });
}, 'init', 'en-gb');

setup('No subfolder path', () => {
    get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi({
        name: 'My Workspace',
        displayName: 'My Workspace'
    }));

}, 'init', 'en-gb');

https://www.gideonpyzer.com/blog/using-es2015-generators-to-mock-api-responses/

Using ES2015 generators to mock API responses

Example: Folder Picker

(() => {
    const popDownTrigger = attr.automation('pop-down-trigger');
    // other constants omitted

    flow('folder-picker-widget', () => {
            step('Go to widget', () => page.gotoWidget('folder-picker-widget'));
            chance({
                "No subfolders": () => {
                    step('Load widget', () => page.runSetup('No subfolder path'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check that neither folders or files are shown', () => {
                        assert.waitWhileVisible(folderList, 'Folder list should not been shown');
                        assert.waitWhileVisible(fileList, 'File list should not been shown');
                    });
                    step('Check for no subfolder text', () => {
                        assert.waitForElementToContainText(leaf, 'This folder is empty', 'Pop down should show "This folder is empty"');
                    });
                },
                "Pick folder, then change mind": () => {
                    step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check folders', () => {
                        myWorkspaceSelected();
                    });
                    step('Click on a file', () => {
                        page.click(fileListItems, () => {
                            myWorkspaceSelected(); // nothing changed because file can't be clicked
                        });
                    });
                    step('Click on File requests', () => {
                        page.click(folderListItems, () => {
                            fileRequestsSelected();
                        });
                    });
                    step('Choose the folder', () => {
                        page.click(chooseFolder, () => {
                            assert.waitForElementToContainText(triggerTitle, 'File requests', 'File requests folder should be shown as selected');
                        });
                    });
                    step('Click on trigger again', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check we are currently in File Requests folder', () => {
                        fileRequestsSelected();
                    });
                    step('Go to previous folder', () => {
                        page.click(goToPrevious, () => {
                            myWorkspaceSelected();
                        });
                    });
                    step('Choose the folder', () => {
                        page.click(chooseFolder, () => {
                            assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'My Workspace folder should be shown as selected');
                        });
                    });
                },
                "Pick a folder, then cancel": () => {
                    step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check folders', () => {
                        myWorkspaceSelected();
                    });
                    step('Click on File requests', () => {
                        page.click(folderListItems, () => {
                            fileRequestsSelected();
                        });
                    });
                    step('Click on cancel', () => {
                        page.click(cancel);
                    });
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                }
            });
        }
    );

    function myWorkspaceSelected() {
        assert.waitForElementToContainText(headerTitle, 'My Workspace', 'My Workspace should be shown for header');
        assert.elementCount(folderListItems, 3, 'There should be three folders');
        assert.waitForNthElementToContainText(folderListItems, 1, 'File requests', 'The first folder should be called File requests');
        assert.elementCount(fileListItems, 2, 'There should be two files');
    }
    function fileRequestsSelected() {
        assert.waitForElementToContainText(headerTitle, 'File requests', 'File requests should be shown for header');
        assert.waitForSelector(chevron, 'Chevron icon should be shown');
        assert.elementCount(folderListItems, 0, 'There should be 0 subfolders');
        assert.elementCount(fileListItems, 1, 'There should be 0 files');
    }
})();

Example: Folder Picker

(() => {
    const popDownTrigger = attr.automation('pop-down-trigger');
    // other constants omitted

    flow('folder-picker-widget', () => {
            step('Go to widget', () => page.gotoWidget('folder-picker-widget'));
            chance({
                "No subfolders": () => {
                    step('Load widget', () => page.runSetup('No subfolder path'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check that neither folders or files are shown', () => {
                        assert.waitWhileVisible(folderList, 'Folder list should not been shown');
                        assert.waitWhileVisible(fileList, 'File list should not been shown');
                    });
                    step('Check for no subfolder text', () => {
                        assert.waitForElementToContainText(leaf, 'This folder is empty', 'Pop down should show "This folder is empty"');
                    });
                },
                "Pick folder, then change mind": () => {
                    step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check folders', () => {
                        myWorkspaceSelected();
                    });
                    step('Click on a file', () => {
                        page.click(fileListItems, () => {
                            myWorkspaceSelected(); // nothing changed because file can't be clicked
                        });
                    });
                    step('Click on File requests', () => {
                        page.click(folderListItems, () => {
                            fileRequestsSelected();
                        });
                    });
                    step('Choose the folder', () => {
                        page.click(chooseFolder, () => {
                            assert.waitForElementToContainText(triggerTitle, 'File requests', 'File requests folder should be shown as selected');
                        });
                    });
                    step('Click on trigger again', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check we are currently in File Requests folder', () => {
                        fileRequestsSelected();
                    });
                    step('Go to previous folder', () => {
                        page.click(goToPrevious, () => {
                            myWorkspaceSelected();
                        });
                    });
                    step('Choose the folder', () => {
                        page.click(chooseFolder, () => {
                            assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'My Workspace folder should be shown as selected');
                        });
                    });
                },
                "Pick a folder, then cancel": () => {
                    step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                    step('Click on trigger', () => {
                        page.click(popDownTrigger, () => {
                            assert.waitForSelector(popDown, 'Folder picker pop down is shown');
                        });
                    });
                    step('Check folders', () => {
                        myWorkspaceSelected();
                    });
                    step('Click on File requests', () => {
                        page.click(folderListItems, () => {
                            fileRequestsSelected();
                        });
                    });
                    step('Click on cancel', () => {
                        page.click(cancel);
                    });
                    step('Check that default workspace is shown', () => {
                        assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
                    });
                }
            });
        }
    );

    function myWorkspaceSelected() {
        assert.waitForElementToContainText(headerTitle, 'My Workspace', 'My Workspace should be shown for header');
        assert.elementCount(folderListItems, 3, 'There should be three folders');
        assert.waitForNthElementToContainText(folderListItems, 1, 'File requests', 'The first folder should be called File requests');
        assert.elementCount(fileListItems, 2, 'There should be two files');
    }
    function fileRequestsSelected() {
        assert.waitForElementToContainText(headerTitle, 'File requests', 'File requests should be shown for header');
        assert.waitForSelector(chevron, 'Chevron icon should be shown');
        assert.elementCount(folderListItems, 0, 'There should be 0 subfolders');
        assert.elementCount(fileListItems, 1, 'There should be 0 files');
    }
})();

1

2

Example: Folder Picker

- Running the develop file

    1. Set the path to use in the develop file

 

 

    2. Set: 

 

- Running the tests

   

if (!window.undertest) {
    setup.run('Happy path');
}
grunt dev:widget --widget=folder-picker
grunt test:widget --test=folder-picker

- Not as nice as Torque for developing and testing, sorry FilesApp

 

- Development:

    - No isolated components/pages,

        - Run:                , open up Huddle and go to FilesApp pages

- Mocks:

    - Can be inconsistent and messy,

    - Often shared - then modifications break other uses

- Tests:

    - Often flaky

    - BUT, they still use Phantom Flow, so that's a win :) 

FilesApp Tests

grunt dev
(function () {
    // constants omitted
    var config = {
        id: fileId,
        user: {
            settings: {
                hasSeenNewUserWebOnBoarding: true,
                isTwoFactorAuthEnabled: false,
                hasViewedWorkspaceOverviewGuidance: true,
                hasViewedFolderGuidance: true,
                hasViewedFilesDetailsGuidance: false,
                hasPerformedNewUserAction: true,
                hasViewedTaskGuidance: false,
                hasViewedFileRequestGuidance: false,
                hasViewedCompanyManagerGuidance: false
            }
        }
    };
    flow('New user files list onboarding', function () {
        step("Setup page", function () {
            huddle.page.setup('file id=' + fileId, goToFileDetailsPage);
        });
        step("Check first coachmark", function () {
            huddle.assert.opaqueAfterWait(firstCoachmark, "Files details modal is visible");
        });
        // other steps omitted
    });
    function goToFileDetailsPage() {
        fileDetailsPage.fakeAllRequests(config);
        huddle.assert.existsAfterWait(fileDetailsPage.body, 'Should have loaded file page');
        casper.evaluate(function () {
            window.localStorage.clear();
        });
    }
}());

Mocking

- Phantom JS is no longer being supported as a headless env.

    - Chromium have worked on a headless mode for Chrome

 

Benefits of Chrome:

- Uses V8, which has a ton of optimisations

    - Parsing and executing JavaScript in tests should be quicker

- It's a real browser

    - Loads of people use it - 64%, according to W3Counter July 17

 

I'm looking into solutions for creating a version of Phantom Flow that uses Chrome via their debugging protocol interface.

 

The Future

Source: https://www.w3counter.com/globalstats.php

Blog:

https://www.gideonpyzer.com/blog/

Github:

https://github.com/gidztech/

Twitter:

@gidztech

Slides:                   

http://slides.com/gidztech/phantomflow#/

 

 

PhantomFlow:     

https://github.com/Huddle/PhantomFlow

PhantomCSS:        

https://github.com/Huddle/PhantomCSS

CasperJS:  

http://casperjs.org/

Resources

PhantomFlow (Huddle Talk)

By Gideon Pyzer

Made with Slides.com

PhantomFlow (Huddle Talk)

UI testing framework based on decision trees

  • 501

More from Gideon Pyzer