Phantom Flow
UI testing framework based on decision trees
"A NodeJS wrapper for PhantomJS, CasperJS and PhantomCSS, PhantomFlow enables a fluent way of describing user flows in code whilst generating structured tree data for visualisation."
"A NodeJS wrapper for PhantomJS, CasperJS and PhantomCSS, PhantomFlow enables a fluent way of describing user flows in code whilst generating structured tree data for visualisation."
Source: https://ifunny.co/fun/hUDAM5X74
Phantom JS
- Headless browser
- Browser runs without a GUI
- Based on WebKit
- The rendering engine used by Safari, and formerly Chrome
- Provides JavaScript API for interacting with the engine and page
var page = require('webpage').create();
page.open('https://www.huddle.com/', function (status) {
if (status !== 'success') {
console.log('Unable to access network');
} else {
var headerText = page.evaluate(function () {
var el = document.querySelector('.hero-text h2');
return el.textContent;
});
console.log(headerText);
}
phantom.exit();
});Example
Casper JS
- Extends the API provided by Phantom JS
- Provides a richer set of interaction functions with the page
- Provides a testing library
var casper = require('casper').create();
casper.start('https://www.huddle.com/', function () {
this.waitUntilVisible('.hero-text h2', function () {
this.test.pass('Header text is visible');
}, function () {
this.test.fail('Header text is not visible');
}, 2000);
});Casper JS API
waitForSelector
waitUntilVisible
waitWhileVisible
click
sendKeys ...
assert
assertEquals
pass
fail
Testing
- Phantom JS - The headless browser
- Casper JS - The interaction and testing library on top
We have the tools to write UI tests.
That's it, right?
Most developers/testers using this tech stack might agree
But we're better than that...
Phantom Flow
- Phantom Flow is a testing framework that allows you to describe user flows, using Decision Trees.
- Tests are usually written in flat, list-like structure
- But humans think/remember better with trees
- Benefits
- Less likely to miss scenarios
- Good way to learn the code base
- Can serve as documentation
Terminology (API)
Flow - The name of the test suite, typically a single component
Step - A single action in the user flow
Decision - A decision point for the user; do action A => some steps; do action B => other steps
Chance - Like a decision but used to describe factors outside the control of the user, e.g. error on the page vs no error on page
Example
flow("Get a coffee", function(){
step("Go to the kitchen", goToKitchen);
step("Go to the coffee machine", goToMachine);
decision({
"Wants Latte": function(){
chance({
"There is no milk": function(){
step("Request Latte", requestLatte_fail);
decision({
"Give up": function(){
step("Walk away from the coffee machine", walkAway);
},
"Wants Espresso instead": wantsEspresso
});
},
"There is milk": function(){
step("Request Latte", requestLatte_success);
}
});
},
"Wants Cappuccino": function(){
chance({
"There is no milk": function(){
step("Request Cappuccino", requestCappuccino_fail);
decision({
"Request Espresso instead": wantsEspresso
});
},
"There is milk": function(){
step("Request Cappuccino", requestCappuccino_success);
}
});
},
"Wants Espresso": wantsEspresso
});
});Assertions
function goToMachine (){
casper.click('#coffeemachinebutton');
casper.waitForSelector(
'#myModal:not([style*="display: none"])',
function success(){
phantomCSS.screenshot('#myModal .modal-content');
casper.test.pass('Should see coffee machine');
},
function timeout(){
casper.test.fail('Should see coffee machine');
}
);
}
casper.test.pass() is an explicit pass for the test, in cases where you don't run standard asserts, such as casper.test.assertEquals()
Assertions
function goToMachine (){
casper.click('#coffeemachinebutton');
casper.waitForSelector(
'#myModal:not([style*="display: none"])',
function success(){
phantomCSS.screenshot('#myModal .modal-content');
casper.test.pass('Should see coffee machine');
},
function timeout(){
casper.test.fail('Should see coffee machine');
}
);
}
What is this crazy Phantom CSS thingy?
Visual Regression
- Unit tests will cover functional code
- We can test whether a value is as expected
- We can test that an element exists on the page
But how do we verify that it looks right?
- We can't easily test that CSS classes and style rules are being applied correctly under the current testing model.
The answer is Phantom CSS!
Phantom CSS
casper.then('https://www.huddle.com/', function () {
phantomCSS.screenshot(selector);
});
- The first time the screenshot function is invoked, the baseline is created. This should be versioned in GIT.
- Subsequent times, a comparison is made (using Resemble.js)
- If the RGB pixel difference is significant enough, the test will fail
- Phantom CSS is a library that compares a baseline screenshot of a component with a new screenshot taken at the time of the test
Phantom CSS
- The first image is the baseline (correct) visual
- The second image is the latest screenshot taken during a test
- The third image shows the differences, outlined in pink
There are position or padding issues indicated here
If you don't like pink, no problem, this be customised :)
Phantom CSS
What if we change the component?
- Failing visual tests don't always mean the component is broken
- We sometimes re-design or move things about
- Solution: Delete the baseline visual
- A new one is generated for versioning next time the test is run
Phantom CSS
Considerations
- Visual tests are not as fast as functional JS tests
- Only use them where necessary, not every element!
- A full page screenshot may not pick up some changes
- The threshold is a percentage of the number of pixel diffs
- Better to be taken at the component level (e.g. widgets)
Dev/Tests Huddle
Torque
- Our component library, made up of widgets and components
- Confused?
- Huddle Terminology:
- Component - UI element(s) with no APIs/data ("dumb")
- Widget - UI element(s), with APIs and data, composed
- Develop our components and widgets in isolation
- Mock APIs and data
- Use our develop environment (page fixtures, mocking) for tests
- Easy to re-produce problems found in tests
Component Dev
- A component will typically have:
- HTML file - the markup
- Knockout JS file - the JavaScript and component binding
- SASS file - CSS with variables, mixins, and other goodies
- YAML config file - Describes inputs, states, implementations
Component Dev
- Describe the inputs for the component
- Describe the implementation, currently always Knockout
Component Dev
- Components are built dynamically using the YAML config file
- Run: to get hot docs (webpack dev server)
- Documentation of our components and all the variations
grunt dev:docs
Component Tests
- Running the tests
- Running the report
- See the baseline image, the new image, and the fail image
- Use Rebase button to replace the baseline image with current
grunt test
grunt test --report=true
Widget Dev
- A widget will typically have:
- HTML file - the markup
- Knockout JS file - the JavaScript and component binding
- SASS file - CSS with variables, mixins, and other goodies
- Develop file - API/data mocks
- Tests folder - with a test file + compiled version
Source: http://blog.interestship.com
Example: Folder Picker
if (!window.undertest) {
setup.run('Happy path');
}
setup('Happy path', () => {
get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi(defaultFolderMock));
get(/\/files\/pagedfolders\/\d+/, function*() {
while(true) {
yield { status:200, response:folderApi(fleRequestsFolderMock) };
}
});
}, 'init', 'en-gb');
setup('Pick File requests folder, then go back to root', () => {
get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi(defaultFolderMock));
get(/\/files\/pagedfolders\/\d+/, function*() {
yield {status:200,response:folderApi(fleRequestsFolderMock)};
while(true) {
yield { status:200, response:folderApi(defaultFolderMock) };
}
});
}, 'init', 'en-gb');
setup('No subfolder path', () => {
get(/\/files\/workspaces\/\d+\/pagedfolders\/root/, folderApi({
name: 'My Workspace',
displayName: 'My Workspace'
}));
}, 'init', 'en-gb');Using ES2015 generators to mock API responses
Example: Folder Picker
(() => {
const popDownTrigger = attr.automation('pop-down-trigger');
// other constants omitted
flow('folder-picker-widget', () => {
step('Go to widget', () => page.gotoWidget('folder-picker-widget'));
chance({
"No subfolders": () => {
step('Load widget', () => page.runSetup('No subfolder path'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check that neither folders or files are shown', () => {
assert.waitWhileVisible(folderList, 'Folder list should not been shown');
assert.waitWhileVisible(fileList, 'File list should not been shown');
});
step('Check for no subfolder text', () => {
assert.waitForElementToContainText(leaf, 'This folder is empty', 'Pop down should show "This folder is empty"');
});
},
"Pick folder, then change mind": () => {
step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check folders', () => {
myWorkspaceSelected();
});
step('Click on a file', () => {
page.click(fileListItems, () => {
myWorkspaceSelected(); // nothing changed because file can't be clicked
});
});
step('Click on File requests', () => {
page.click(folderListItems, () => {
fileRequestsSelected();
});
});
step('Choose the folder', () => {
page.click(chooseFolder, () => {
assert.waitForElementToContainText(triggerTitle, 'File requests', 'File requests folder should be shown as selected');
});
});
step('Click on trigger again', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check we are currently in File Requests folder', () => {
fileRequestsSelected();
});
step('Go to previous folder', () => {
page.click(goToPrevious, () => {
myWorkspaceSelected();
});
});
step('Choose the folder', () => {
page.click(chooseFolder, () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'My Workspace folder should be shown as selected');
});
});
},
"Pick a folder, then cancel": () => {
step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check folders', () => {
myWorkspaceSelected();
});
step('Click on File requests', () => {
page.click(folderListItems, () => {
fileRequestsSelected();
});
});
step('Click on cancel', () => {
page.click(cancel);
});
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
}
});
}
);
function myWorkspaceSelected() {
assert.waitForElementToContainText(headerTitle, 'My Workspace', 'My Workspace should be shown for header');
assert.elementCount(folderListItems, 3, 'There should be three folders');
assert.waitForNthElementToContainText(folderListItems, 1, 'File requests', 'The first folder should be called File requests');
assert.elementCount(fileListItems, 2, 'There should be two files');
}
function fileRequestsSelected() {
assert.waitForElementToContainText(headerTitle, 'File requests', 'File requests should be shown for header');
assert.waitForSelector(chevron, 'Chevron icon should be shown');
assert.elementCount(folderListItems, 0, 'There should be 0 subfolders');
assert.elementCount(fileListItems, 1, 'There should be 0 files');
}
})();
Example: Folder Picker
(() => {
const popDownTrigger = attr.automation('pop-down-trigger');
// other constants omitted
flow('folder-picker-widget', () => {
step('Go to widget', () => page.gotoWidget('folder-picker-widget'));
chance({
"No subfolders": () => {
step('Load widget', () => page.runSetup('No subfolder path'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check that neither folders or files are shown', () => {
assert.waitWhileVisible(folderList, 'Folder list should not been shown');
assert.waitWhileVisible(fileList, 'File list should not been shown');
});
step('Check for no subfolder text', () => {
assert.waitForElementToContainText(leaf, 'This folder is empty', 'Pop down should show "This folder is empty"');
});
},
"Pick folder, then change mind": () => {
step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check folders', () => {
myWorkspaceSelected();
});
step('Click on a file', () => {
page.click(fileListItems, () => {
myWorkspaceSelected(); // nothing changed because file can't be clicked
});
});
step('Click on File requests', () => {
page.click(folderListItems, () => {
fileRequestsSelected();
});
});
step('Choose the folder', () => {
page.click(chooseFolder, () => {
assert.waitForElementToContainText(triggerTitle, 'File requests', 'File requests folder should be shown as selected');
});
});
step('Click on trigger again', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check we are currently in File Requests folder', () => {
fileRequestsSelected();
});
step('Go to previous folder', () => {
page.click(goToPrevious, () => {
myWorkspaceSelected();
});
});
step('Choose the folder', () => {
page.click(chooseFolder, () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'My Workspace folder should be shown as selected');
});
});
},
"Pick a folder, then cancel": () => {
step('Load widget', () => page.runSetup('Pick File requests folder, then go back to root'));
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
step('Click on trigger', () => {
page.click(popDownTrigger, () => {
assert.waitForSelector(popDown, 'Folder picker pop down is shown');
});
});
step('Check folders', () => {
myWorkspaceSelected();
});
step('Click on File requests', () => {
page.click(folderListItems, () => {
fileRequestsSelected();
});
});
step('Click on cancel', () => {
page.click(cancel);
});
step('Check that default workspace is shown', () => {
assert.waitForElementToContainText(triggerTitle, 'My Workspace', 'The default workspace is shown');
});
}
});
}
);
function myWorkspaceSelected() {
assert.waitForElementToContainText(headerTitle, 'My Workspace', 'My Workspace should be shown for header');
assert.elementCount(folderListItems, 3, 'There should be three folders');
assert.waitForNthElementToContainText(folderListItems, 1, 'File requests', 'The first folder should be called File requests');
assert.elementCount(fileListItems, 2, 'There should be two files');
}
function fileRequestsSelected() {
assert.waitForElementToContainText(headerTitle, 'File requests', 'File requests should be shown for header');
assert.waitForSelector(chevron, 'Chevron icon should be shown');
assert.elementCount(folderListItems, 0, 'There should be 0 subfolders');
assert.elementCount(fileListItems, 1, 'There should be 0 files');
}
})();
1
2
Example: Folder Picker
- Running the develop file
1. Set the path to use in the develop file
2. Set:
- Running the tests
if (!window.undertest) {
setup.run('Happy path');
}grunt dev:widget --widget=folder-picker
grunt test:widget --test=folder-picker
- Not as nice as Torque for developing and testing, sorry FilesApp
- Development:
- No isolated components/pages,
- Run: , open up Huddle and go to FilesApp pages
- Mocks:
- Can be inconsistent and messy,
- Often shared - then modifications break other uses
- Tests:
- Often flaky
- BUT, they still use Phantom Flow, so that's a win :)
FilesApp Tests
grunt dev
(function () {
// constants omitted
var config = {
id: fileId,
user: {
settings: {
hasSeenNewUserWebOnBoarding: true,
isTwoFactorAuthEnabled: false,
hasViewedWorkspaceOverviewGuidance: true,
hasViewedFolderGuidance: true,
hasViewedFilesDetailsGuidance: false,
hasPerformedNewUserAction: true,
hasViewedTaskGuidance: false,
hasViewedFileRequestGuidance: false,
hasViewedCompanyManagerGuidance: false
}
}
};
flow('New user files list onboarding', function () {
step("Setup page", function () {
huddle.page.setup('file id=' + fileId, goToFileDetailsPage);
});
step("Check first coachmark", function () {
huddle.assert.opaqueAfterWait(firstCoachmark, "Files details modal is visible");
});
// other steps omitted
});
function goToFileDetailsPage() {
fileDetailsPage.fakeAllRequests(config);
huddle.assert.existsAfterWait(fileDetailsPage.body, 'Should have loaded file page');
casper.evaluate(function () {
window.localStorage.clear();
});
}
}());Mocking
- Phantom JS is no longer being supported as a headless env.
- Chromium have worked on a headless mode for Chrome
Benefits of Chrome:
- Uses V8, which has a ton of optimisations
- Parsing and executing JavaScript in tests should be quicker
- It's a real browser
- Loads of people use it - 64%, according to W3Counter July 17
I'm looking into solutions for creating a version of Phantom Flow that uses Chrome via their debugging protocol interface.
The Future
Blog:
https://www.gideonpyzer.com/blog/
Github:
Twitter:
@gidztech
Slides:
http://slides.com/gidztech/phantomflow#/
PhantomFlow:
https://github.com/Huddle/PhantomFlow
PhantomCSS:
https://github.com/Huddle/PhantomCSS
CasperJS: