E2E testing with Puppeteer

Organised by:

PUG Milano

Hosted by:

by Stefano Magni (@NoriSte)

Front-end Developer

Level: basic
Prerequisites: knowledge of JS Promises and async/await

23/01/2019

If you wanna play alone:

• clone/download

https://github.com/NoriSte/grusp-talk-e2e-testing-with-puppeteer

• $ npm install

• $ npm test

If everything works the end result will be:

 PASS  test/test-4.e2e.test.js (5.024s)
 PASS  test/test-5.e2e.test.js (5.412s)
 PASS  test/test-1.e2e.test.js (8.213s)
 PASS  test/test-2.e2e.test.js (9.121s)
 PASS  test/test-3.e2e.test.js (11.316s)
 PASS  test/test-6.test.js (20.531s)

https://slides.com/noriste/e2e-testing-talk-grusp

This night's slides

Long story short:

at the beginning there was the callback hell...

What is async/await?

function logAll(){
  logWithCallback("A", () => {
    logWithCallback("B", () => {
      logWithCallback("C", () => {
        logWithCallback("D", () => {});
      });
    });
  });
}
logAll();

... Then the promises come...

• better readability

• better control

• easier error catching

What is async/await?

function logAll(){
  logWithPromise("A")
  .then(() => logWithPromise("B"));
  .then(() => logWithPromise("C"));
  .then(() => logWithPromise("D"));
}
logAll();

... Then you can now await a promise...

• even better readability

• more concise code

What is async/await?

async function logAll(){
  await logWithPromise("A"));
  await logWithPromise("B"));
  await logWithPromise("C"));
  await logWithPromise("D"));
}
logAll();

from callbacks...

What is async/await?

async function logAll(){
  await logWithPromise("A");
  await logWithPromise("B");
  await logWithPromise("C");
  await logWithPromise("D");
}
logAll();

function logAll(){
  logWithCallback("A", () => {
    logWithCallback("B", () => {
      logWithCallback("C", () => {
        logWithCallback("D", () => {});
      });
    });
  });
}
logAll();

... to waiting promises

Jest is a zero-configuration testing framework.

What is Jest?

https://jestjs.io

beforeAll(async () => {
  // code to be run before the whole test suite
});
beforeEach(async () => {
  // code to be run before every test
});
afterAll(async () => {
  // code to be run after the whole test suite
});

describe('Description 1', () => {
  test('Test 1', () => {
      expect(1).toBe(1);
  });
  test('Test 2', () => {
      expect('hello').not.toBe('world');
  });

  describe('Inner description', () => {
    test('Inner test 1', () => {
        expect({foo: 'bar'}).toEqual({foo: 'bar'});
    });
    test('Inner test 2', () => {
        expect(0).toBeFalsy();
    });
  });
});

 PASS  ./test.js
  Description 1
    ✓ Test 1 (5ms)
    ✓ Test 2 (1ms)
    Inner description
      ✓ Inner test 1 (2ms)
      ✓ Inner test 2 (1ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        1.071s
Ran all test suites.

test.js

result of $ jest

What is E2E testing

(or UI testing)?

From Wikipedia:

End-to-end testing is a methodology used to test whether the flow of an application is performing as designed from start to finish.

It is useful to test your site/webapp/app in the same way your real users consume it.

The user interactions are automated.

From the Puppeteer.page.click API documentation:

This method fetches an element with selector, scrolls it into view if needed, and then uses page.mouse to click in the center of the element.

E2E tests are the only tool that can check if your website really works or not.

The most important thing:

E2E tests are hosted in a real browser.

What is Puppeteer?

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.

Puppeteer lets you control a Chrome instance and fire user-like interactions (click, typing etc.) within the page.

https://github.com/GoogleChrome/puppeteer

It introduces the concept of headless browser that is a full and standard browser that runs without a GUI.

1: Node running the script

2: Chromium launched with Puppeteer

3: Code executed in page

A typical script to launch Puppeteer

The button takes the user to another page.

Complete the test that checks it.

git checkout test-1

open ./dist/test-1.html

open ./test/test-1.e2e.test.js

• index.html has a big blue button

• test-1.e2e.test.js misses something (see the comments at line 22)

• launch the test suite with $ npm test

Test 1

describe(`That's our first E2E test`, () => {
  test(`The button brings the user to the next page`, async () => {
    await page.goto(`file:${path.join(__dirname, './../dist/test-1.html')}`);

    // always add a 'data-test' attribute to the elements that will
    // participate to your tests
    await page.click('[data-test="button"]');

    // checking for a specific content is a good way to be 100% sure
    // that the page has been loaded
    await expect(page).toMatch('Hello from PUG MI');
  }, 5000);
});

Solution

Remember some useful characteristics of Puppeteer...

You can install Chrome extensions on launch.

// welcome external devtools
return await puppeteer.launch({
  args: [
    // loading Chrome Vue devtools
    `--load-extension=/.../vue-devtools/shells/chrome/`,
    ]
});

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#working-with-chrome-extensions

You can launch a single and shareable Chrome instance to improve performance.

// if you have previously saved the Chrome endpoint you can connect to it
// instead of creating a new one
browser = await puppeteer.connect({browserWSEndpoint: existingEndPoint});

e.g. https://github.com/smooth-code/jest-puppeteer

You can go incognito.

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#browsercreateincognitobrowsercontext

const browser = await puppeteer.launch();
// Create a new incognito browser context.
const context = await browser.createIncognitoBrowserContext();
// Create a new page in a pristine context.
const page = await context.newPage();
// Do stuff
await page.goto('https://example.com');

It allows you to easily understand if there are errors in page.

const browser = await puppeteer.launch({});
const page = await browser.newPage();

// when an error is thrown in Chrome
// it's logged by the node process
page.on('pageerror', e => console.log(e.text))

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#event-pageerror

You can intercept requests.

await page.setRequestInterception(true);

page.on('request', request => {
  // it intercepts every call to register.php and
  // responds with a custom JSON
  if (request.url() === 'register.php') {
    request.respond({
      content: 'application/json',
      body: JSON.stringify({registered: true})
    });
  }
  else {
    // other requests work as usual
    request.continue();
  }
});

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#event-request

You can set cookies.

await page.setCookie({
    name: 'loggedIn',
    value: '1'
});

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetcookiecookies

You can emulate devices.

const device = {
    'userAgent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1',
    'viewport': {
      'width': 375,
      'height': 667,
      'isMobile': true,
      'hasTouch': true,
      'isLandscape': false
    }
  }
await page.emulate(device);

// or you can use one of the presets

const devices = require('puppeteer/DeviceDescriptors');
const iPhone6 = devices['iPhone 6'];
await page.emulate(iPhone6);

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageemulateoptions

• $, $$, $eval, $$eval: DOM utilities

• addScriptTag, addStyleTag: add resources in page

• page.evaluate: runs a function into the browser

• page.exposeFunction: adds a function

• page.click

• page.hover

And many others

https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md

How can we debug a Puppeteer script?

Turn off headless mode

So you can see what the browser is doing.

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

const browser = await puppeteer.launch({
    headless: false
});

Slow down the browser

Not ideal for very long tests but super useful though.

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

const browser = await puppeteer.launch({
  headless: false,
  slowMo: 250 // slow down every action by 250ms
});

Avoid closing the browser

afterAll(async () => {
  // await browser.close();
});

Directly from... me 😊

Make a screenshot if a test fails. It can help you avoid relaunching the suite with the browser in non-headless mode.

Make screenshots

if (array.length !== 3)
    await page.screenshot({path: 'screenshot.png'})
expect(array.length).toBe(3)

Capture console output

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

page.on('console', msg => console.log('PAGE LOG:', msg.text()));

Launch Puppeteer with the DevTools already opened

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

const browser = await puppeteer.launch({
  headless: false,
  slowMo: 250, // slow down by 250ms
  devtools: true
});

Add an evaluate statement with debugger inside

Or add debugger to an existing evaluate statement (it works if the Chrome DevTools are opened).

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

await page.evaluate(() => {debugger;});
// The test will now stop executing in the above evaluate statement,
// and chromium will stop in debug mode.

Change default test timeout

Directly from

https://github.com/GoogleChrome/puppeteer#debugging-tips

jest.setTimeout(100000);

Run a single test/suite

describe.only('Temporary run only me please', () => {
    test('The test we want to isolate', () => { ... });
});

describe('A suite that won\'t be launched', () => {
    test('Test 2', () => { ... });
});

describe('Another suite that won\'t be launched', () => {
    test('Test 3', () => { ... });
});

From Jest

https://jestjs.io/docs/en/api#describeonlyname-fn

Skip some tests/suites

describe('Suite 1', () => {
    test('The test we want to isolate', () => { ... });
});

describe.skip('A suite that will be skipped', () => {
    test('Test 2', () => { ... });
});

describe('Suite 3', () => {
    test.skip('A test that will be skipped', () => { ... });
});

From Jest

https://jestjs.io/docs/en/api#describeskipname-fn

console.log (into the Chrome instance) the name of the test, so you always know which test is running if you are watching them.

await page.evaluate(() => console.log('Test name'));

Directly from... me 😊

You can also add a fixed div in page to show the test name at the top of the page, it’s not ideal but it could be very useful in some situations.

Directly from... me 😊

Use the right assertion

https://jestjs.io/docs/en/expect

Directly from... me 😊

Expected: true
Received: false
    3 | describe(`Test`, () => {
    4 |   test(`Leverage assertions`, () => {
  > 5 |     expect(getObj().key === 5).toBe(true);
// Ok, what is the value of getObj().key?

Expected: 5
Received: 4
    3 | describe(`Test`, () => {
    4 |   test(`Leverage assertions`, () => {
  > 5 |     expect(getObj().key).toBe(5);
// ok I'll fix it without relaunching the suite

It should be obvious but remember: the fact that your suite tests is running in a real and interactive browser... it doesn't mean that you should interact with it!

When you are running the browser in non-headless mode remember to not affect the test with your mouse/keyboard.

Never interact with the test

<a data-test="button">

<!--
find it with the following selector
[data-test="button"]
-->

Always add a data-test (or data-testid) attribute to the elements that will be referenced by the tests.

Best practice

Why?

Because every attribute has a “role”.

We use .classes for CSS (and they got replaced in case of CSS Modules), #ids for JS, and therefore they can change based on CSS and JS needs…

A dedicated attribute is more resilient to refactoring and leads every (diligent) developer to keep it or, at least, ask himself for the attribute use.

data-test attribute

Let’s speak about the cons!

They’re the longest to be written, not in terms of lines of code but in terms of variables to be considered.

Cons

Why?

Because you aren’t in isolation (like in Unit tests), nor in a super-mocked environment...

You have a real browser on a real network, you’ll face network latencies, temporary downs of the server, unpredictable AD scripts and banners, (possible) browser drivers inconsistencies…

Cons

Simulating the (exact) user behaviour sometimes could be very tricky (blur, input typos etc.).

Cons

If you try to DRY your tests you’re probably adding another (hard to manage) complexity layer.

Cons

They are slow, even with almost instant UI interactions.

Cons

It's the same of the first test but a the page has a cookie footer...

Test 2

describe(`That's our second E2E test`, () => {
  beforeAll(async () => {
    await page.goto(`file:${path.join(__dirname, './../dist/test-2.html')}`);

    // don't let the test fail for a silly element like a cookie footer
    // It could be already accepted when you navigate to another page
    if(await page.$('[data-test="cookie-footer-acceptance"]')) {
      try {
        // what happens if it exists but isn't clickable (eg. it's hidden)?
        // A try/catch will manage the case
        await page.click('[data-test="cookie-footer-acceptance"]');
      } catch(e) {
        // the element exists but isn't clickable
      }
    }
  });
});

Solution

Never use some "sleep" code, you can’t determine how much a page/script could be waited for (render waitings, network conditions etc.).

Use waiters, promises, framework-specific render callbacks (like Vue.nextTick) but not sleep.

Never "sleep"

Again! Now the cookie footer disappears with a CSS animation!

Test 3

if(await page.$('[data-test="cookie-footer-acceptance"]')) {
  try {
    await page.click('[data-test="cookie-footer-acceptance"]');
    // you can wait that an element is hidden
    // @see https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitforselectorselector-options
    await page.waitForSelector('#cookie-footer', {
      hidden: true
    });
  } catch(e) {
    // the element exists but maybe it isn't clickable
  }
}

Solution

With page.evaluate you can execute code into the running browser (like you do when you write code into the DevTools console) and give back the result to your NodeJS script.

Page.evaluate

Page.evaluate example

const puppeteer = require('puppeteer');

(async (options) => {
  browser = await puppeteer.launch({
    headless: false,
    devtools: true
  });
  page = await browser.newPage();

  console.log('Node logging');
  await page.evaluate(
    () => console.log('Browser logging')
  );
})()

const seven = 7;
const result = await page.evaluate(aNumber => {
  // JQuery is in page? Redux?
  // Use them and give back results to the NodeJS script
  return aNumber * 10;
}, seven);
console.log(result); // 70

Take care of the scope

const seven = 7;
const result = await page.evaluate(() => {
    // error! Because the scope is the Chrome's window!
    // Not the NodeJS script!
    return seven * 10;
});
console.log(result);

To better understand the scope problem: remember that the Node script and the launched Chrome instance are separated

Use a promise for every async code

const seven = 7;
const result = await page.evaluate(aNumber => {
  // you can use a promise for every async stuff
  return new Promise(resolve => {
    setTimeout(() => resolve(aNumber*10), 1000);
  });
}, seven);
console.log(result); // 70

About the test-4 solution

// code refactored without arrow functions
await page.evaluate(function() {
  // a Promise is returned to the page.evaluate
  // so it waits until the promise is fullfilled
  return new Promise(function(resolve) {
    // now we're in the Chromium instance, we can listen for
    // the event triggered on the window
    window.addEventListener('cookieFooterDidHide', function(){
      // when the event is been triggered we fullfill the promise
      resolve();
    });
  });
});
// the code after the page.evaluate will be run once the event
// in the browser will be triggered

Page.evaluate is often used to retrieve data from the DOM

const SELECTOR = '[href]:not([href=""])';
let link;

link = await page.evaluate((sel) => 
    document.querySelector(sel).getAttribute('href')
  , SELECTOR);

But it isn't the only way to do that!

const SELECTOR = '[href]:not([href=""])';
let link;

link = await page.evaluate((sel) => 
    document.querySelector(sel).getAttribute('href')
  , SELECTOR);

// compare the two following examples

link = await page.$eval(SELECTOR, el => el.getAttribute('href'));

// or

link = await page.$(SELECTOR).getProperty('href').jsonValue();

The last time with the button, trust me 😊

Now the cookie footer disappears (from a user perspective... not from a CSS one) dispatching an event!

Test 4

if(await page.$('[data-test="cookie-footer-acceptance"]')) {
  try {
    await page.click('[data-test="cookie-footer-acceptance"]');

    // @see https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageevaluatepagefunction-args
    await page.evaluate(() => new Promise(resolve => {
      // the following code will run into the browser page
      window.addEventListener('cookieFooterDidHide', resolve);
    }));

  } catch(e) {
    // the element exists but maybe it isn't clickable
  }
}

Solution

What should I test?

I suggest you to test only the happy paths or the ones that generate value ($$$) for your company.

Test something that broke in the past or that breaks often.

Test 5

• index.html is the usual Todo List app

• add all the todos in the array, remove the first two and then... read the store and check that the last two todos exist in the store itself.

P.s. I exposed a window.vueInstance variable so you can access the store with window.vueInstance.$store.state.todos

# TLDR
$ open test-5.e2e.test.js

Solution

This test leaves room for a series of cases that should make you think about how it’s difficult to manage all the cases consistently when E2E testing.

Others best practices

Puppeteer is Chrome-only (at the moment) but if you use Selenium etc. remeber to avoid launching every test on every browser. Choose a reference browser and carefully select the test to be run on the other ones.

Choose where to test

In a page where everything changes soon… try to standardize your testing environment to avoid false negatives.

Attention to the pages that update frequently

Don't think to test every corner case with E2E testing, it's pure madness.

Leave perfectionism at home

Puppeteer alternatives?

widely used

not so easy to install/setup

it has a WebDriver for every desktop browser (IE too)

Expected Conditions FTW (a sort of Puppeteer's page.waitForSelector on steroids)

Selenium WebDriver

amazing Command Log UI

clear errors

play/pause functionality

made only for E2E testing, it's not a generic automation browser

Cypress.io

works in real browsers (so you can even test on mobile ones)

supported by BrowserStack and CrossBrowserTesting

who uses it loves it 🙂

TestCafe

Again, first of all: consider what and why you should test on mobile...

You can use the device emulation of Chrome DevTools with Puppeteer

If you really need to test on real devices choose TestCafe

What about mobile browsers?

If you want to see some differences between Puppeteer/TestCafe/Cypress you can take a look at a repository I made to solve an issue on StackOverflow:

https://github.com/NoriSte/stackoverflow-52383438-cypress-issue

PHP-friendly libraries

• Puphpeteer https://github.com/nesk/puphpeteer

• Symfony Panther https://github.com/symfony/panther

Are E2E tests framework-agnostic?

Yes but every framework has (obviously) a lot of libraries to help you write them.

What can I do with Puppeteer as well as test my web apps?

Automate annoying and repetitive browser tasks

We used it to configure a new website with ISPConfig.

Scrape whatever you want from the web (eg. data fetch without APIs)

Use it as a development tool

If you're developing a "change password" flow, for every change you make to the code you need to manually:

• login

• go to the profile page

• fill the form to change the password

• click logout

• login with the new password

Use it as a development tool

just to realize that you have a bug... Then fix it and start again...

You could use Puppeteer to bring you directly at the end of the flow.

And once you finished... Add some assertions and your E2E test is ready 😊

Use it while studying

When you're studying a new framework you don't know if you're breaking something you developed just some hours ago... unless you check it manually! Use some tests instead.

Eg. I used them extensively during my first refactors while studying Vue.

Scrape the first 30 results from Google for the given query, have fun 😊

Test 6

Solution

# TLDR
$ open test-6.e2e.test.js

// you have two ways for retrieving an element attribute,
// one is through JSHandle functions...
const jsHandle = await els[j].getProperty('href');
const url = await jsHandle.jsonValue();
// ...
// ...
// ... and one with the usual page.evaluate function
const url = await page.evaluate(
  (sel) => document.querySelector(sel).getAttribute('href'),
nextButtonSelector);

Solution

If we have enough time...
Test redux-saga repo

Test 7

https://github.com/NoriSte/redux-saga-login-react-example

• E2E tests really matter, they don't know anything about your architecture, they test what the end user sees

• they’re quite easy to develop, with a flat learning curve

• write a few amount of tests, don't delegate to E2E testing something you can test with other testing methodologies

Conclusion

• remember to take the tests simple, they could be extremely time-consuming

• never "sleep" the browser

• use "data-test"/"data-testid" attributes

• remember that you can scrape/execute whatever you want on the web with an automated browser

Conclusion

• https://puppeteersandbox.com

Playground

• https://www.blazemeter.com/blog/top-15-ui-test-automation-best-practices-you-should-follow

• https://medium.freecodecamp.org/why-end-to-end-testing-is-important-for-your-team-cb7eb0ec1504

• https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c

• https://hackernoon.com/testing-your-frontend-code-part-iii-e2e-testing-e9261b56475

• https://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting-yourself-in-the-foot-2/

• https://willowtreeapps.com/ideas/how-to-get-the-most-out-of-ui-testing

• http://www.softwaretestingmagazine.com/knowledge/graphical-user-interface-gui-testing-best-practices/

• https://www.slideshare.net/Codemotion/codemotion-webinar-protractor

• https://frontendmasters.com/courses/testing-javascript/introducing-end-to-end-testing/

• https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3

• https://medium.com/yld-engineering-blog/evaluating-cypress-and-testcafe-for-end-to-end-testing-fcd0303d2103

Sources

• https://egghead.io/courses/end-to-end-testing-with-google-s-puppeteer-and-jest



• https://egghead.io/courses/end-to-end-testing-with-cypress

Egghead courses

• https://testingjavascript.com/

Kent's course (reccomended)

• PUG MI for all the work they do

• TAG for the amazing location

• Kent C. Dodds for the amazing “Solidifying what you learn” post

• Luca Previtali and Creeo Studio that allowed me to try this talk in advance

A special thank to:

Thank you!

by Stefano Magni (@NoriSte)

Front-end Developer

The repository with the code and the link to these slides

https://github.com/NoriSte/grusp-talk-e2e-testing-with-puppeteer

Organised by:

Grusp

Hosted by:

23/01/2019