Building confidence in times of change

using automation

By

Shadi Sharaf

About me

<role>

  Software engineer</role>

<ul id="experience">

  <li id="years">

    17+ years of experience 👨🏼‍🦳</li>

  <li id="passion">

    Software engineering, standards, and high performing teams.</li>

  <li id="worked-for">

    The Times, Human Made, XWP, and more.</li>

  <li id="worked-with">

    Google, Siemens, Sony, Disney, and more.</li>

  <li id="worked-on">

    Stream, WordPress Coding Standards</li>

</ul>

Agenda

  • The challenge of Regression
  • How does it happen
  • What does that mean
  • What can we do about
  • Introduction to Playwright
  • Generating automated tests
  • Performance benchmarking

Regression {noun}

A change in the behavior of a system that is not expected.

Challenge: Regression

How does that happen?

Code changes can have unexpected consequences

Challenge: Regression

Why is that dangerous?

What you don’t know can still burn.

Challenge: Regression

What can we do about it?

Don’t wait for the smoke

Challenge: Regression

Active testing

aka QA

TEST ALL THE THINGS!

Challenge: Regression

Active testing

aka QA

BUT....

Challenge: Regression

Passive monitoring

Don’t leave problems in the dark

Challenge: Regression

Playwright {noun}

A person who writes plays.

Solution: Playwright

Why Playwright?

- Cross-browser testing

- Modern architecture

- Powerful API

- Flexible Emulation

- Great developer experience

- WordPress integration

Solution: Playwright

Setting up

$ yarn create playwright # OR npm init playwright

✔ Do you want to use TypeScript or JavaScript? · JavaScript
✔ Where to put your end-to-end tests? · tests
✔ Add a GitHub Actions workflow? (y/N) · false
✔ Install Playwright browsers (can be done manually via 'yarn playwright install')? (Y/n) · true

Writing playwright.config.js.
Writing tests/example.spec.js.
Writing tests-examples/demo-todo-app.spec.js.
Writing package.json.
Downloading browsers (yarn playwright install)…
✔ Success! Created a Playwright Test project at /playwright/stage

Happy hacking! 🎭

Solution: Playwright

Running tests

$ yarn playwright test

Running 6 tests using 4 workers
  6 passed (5.7s)

To open last HTML report run:

  yarn playwright show-report

Solution: Playwright

Viewing reports

Solution: Playwright

Test {noun}

The ultimate prank you play on your code—surprising it with tricky questions, goofy conditions, and unexpected hurdles just to see if it slips on a banana peel or stands tall and proud

Solution: Playwright - Generating tests

Anatomy of a test

import { test, expect } from '@playwright/test';

test('has title', async ({ page }) => {
  // Go to the URL
  await page.goto('https://playwright.dev/');

  // Expect a title "to contain" a substring.
  await expect(page).toHaveTitle(/Playwright/);
});

test('get started link', async ({ page }) => {
  // Go to the URL
  await page.goto('https://playwright.dev/');

  // Click the get started link.
  await page.getByRole('link', { name: 'Get started' }).click();

  // Expects page to have a heading with the name of Installation.
  await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

Solution: Playwright - Generating tests

1. Arrange

2. Act

3. Assert

Solution: Playwright - Generating tests

Locators 🔍

// Locate an element by explicit accessibility role
const element = await page.getByRole('link', { name: 'Get started' });

// Locate an element by text content
const element = await page.getByText('Get started');

// Locate an element by aria-label
const element = await page.getByLabel('Search');

// Locate an element by placeholder
const element = await page.getByPlaceholder('Search');

// Locate an element by alt text
const element = await page.getByAltText('Playwright logo');

// Locate an element by title
const element = await page.getByTitle('Playwright logo');

// Locate an element by data-testid
const element = await page.getByTestId('my-element');

Solution: Playwright - Generating tests

Actions 🔨

// Click the element
element.click();

// Focus the element
element.focus();

// Fill the form field
element.fill();

// Check and uncheck a checkbox
element.check();
element.uncheck();

// Select option in the dropdown
element.selectOption();

// Press single key
element.press();

// Upload files
element.setInputFiles();

Solution: Playwright - Generating tests

Assertions 🧐

// Check if the element is visible
await expect(element).toBeVisible();

// Check if the element is disabled
await expect(element).toBeDisabled();

// Check if the checkbox is checked
await expect(element).toBeChecked();

// Check if the element is focused
await expect(element).toBeFocused();

// Check if the element contains text
await expect(element).toContainText('Hello');

// Check if the element has an attribute
await expect(element).toHaveAttribute('href', 'https://playwright.dev/');

// Check if the element has a value
await expect(element).toHaveValue('Hello');

Solution: Playwright - Generating tests

Visual regressions 🤯

// Generate OR check the screenshot
await expect(element).toHaveScreenshot();

Solution: Playwright - Generating tests

Too much?

Too technical?

Solution: Playwright - Generating tests

Let's
Go to
The Fun part!

Solution: Playwright - Generating tests

Generating tests

Solution: Playwright - Generating tests

Generating tests

$ yarn playwright codegen https://playwright.dev/

Solution: Playwright - Generating tests

Generating tests

Solution: Playwright - Generating tests

Generating tests

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await page.getByRole('link', { name: 'Get started' }).click();
  await page.getByRole('link', { name: 'Generating tests' }).click();
  await expect(page.locator('h1')).toContainText('Generating tests');
  await expect(page.getByRole('article')).toMatchAriaSnapshot(`
    - list:
      - listitem:
        - link "How to record a test"
      - listitem:
        - link "How to generate locators"
    `);
});

Solution: Playwright - Generating tests

Running Tests

Solution: Playwright - Generating tests

Running Tests

$ yarn playwright test

Solution: Playwright - Generating tests

Running Tests

$ yarn playwright test

Running 6 tests using 4 workers
  6 passed (6.7s)

To open last HTML report run:

  yarn playwright show-report

Solution: Playwright - Generating tests

Running Tests (UI mode)

 

$ yarn playwright test --ui

Solution: Playwright - Generating tests

Running Tests (UI mode)

 

Solution: Playwright - Generating tests

Running Tests (Debug mode)

 

$ yarn playwright test --debug

Solution: Playwright - Generating tests

Running Tests (Debug mode)

 

Solution: Playwright - Generating tests

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await page.getByRole('link', { name: 'Get started' }).click();
  await page.getByRole('link', { name: 'Generating tests' }).click();
  await expect(page.locator('h1')).toContainText('Generating toasts');
  await expect(page.getByRole('article')).toMatchAriaSnapshot(`
    - list:
      - listitem:
        - link "How to record a test"
      - listitem:
        - link "How to generate locators"
    `);
});

Running Tests (Debug mode) 🐞

 

Solution: Playwright - Generating tests

Running Tests (Debug mode) 🐞

 

Solution: Playwright - Generating tests

Traces

Solution: Playwright - Generating tests

Traces

$ yarn playwright test --trace on

Solution: Playwright - Generating tests

Traces

Solution: Playwright - Generating tests

Questions?

Solution: Playwright - Generating tests

Performance
{noun}

When your code takes center stage and delivers a flawless show—no hiccups, just pure applause

Solution: Playwright - Performance

Why measure Performance?

less time = more users

more users = more money

more money = less work

less work = more ✈️🏖️ time

Solution: Playwright - Performance

Why measure Performance  

with Playwright?

- Native Performance APIs

- CDP APIs

Solution: Playwright - Performance

Navigation

Resources

Paint

Solution: Playwright - Performance

Navigation

const pageTime = performance.getEntriesByType('navigation');
{
  // ..
  "duration": 1854.7000000029802, // The difference between startTime and responseEnd
  "fetchStart": 11.200000002980232, // Time at which the fetch starts
  "domainLookupStart": 13.100000008940697, // Time at which the domain lookup starts
  "domainLookupEnd": 13.100000008940697, // Time after the domain lookup
  "connectStart": 13.100000008940697, // Time before establishing a server connection
  "secureConnectionStart": 44.80000001192093, // Time before the SSL handshake
  "connectEnd": 269.80000001192093, // Time after establishing a server connection
  "requestStart": 269.90000000596046, // Time before the browser requests the resource
  "responseStart": 1293.4000000059605, // Time when the browser receives the first byte of data
  "responseEnd": 1298, // Time after receiving the last byte or closing the connection
  "transferSize": 10698, // The resource size in bytes including the header and compressed body
  "domInteractive": 1475, // Time at which the document becomes interactive
  "domContentLoadedEventStart": 1475.1000000089407, // Time before document’s DOMContentLoaded event fires
  "domContentLoadedEventEnd": 1520, // Time after document’s DOMContentLoaded event completes
  "domComplete": 1851.9000000059605, // Time at which the document becomes complete
  "loadEventStart": 1851.9000000059605, // Time at which the window.onload event fires
  "loadEventEnd": 1854.7000000029802, // Time after the window.onload event completes
  // ..
}

Solution: Playwright - Performance

Resources

const resourceTime = performance.getEntriesByType('resource');
  {
    connectEnd: 195,
    connectStart: 195,
    decodedBodySize: 0,
    domainLookupEnd: 195,
    domainLookupStart: 195,
    duration: 2,
    encodedBodySize: 0,
    entryType: "resource",
    fetchStart: 195,
    initiatorType: "script",
    name: "https://test.com/script.js",
    requestStart: 195,
    responseEnd: 197,
    responseStart: 197,
    secureConnectionStart: 195,
    serverTiming: [],
    startTime: 195,
    transferSize: 0,
    workerStart: 195
  }

Solution: Playwright - Performance

Paint

const webVitals = performance.getEntriesByType('paint');
[
  {
    "name": "first-paint",
    "entryType": "paint",
    "startTime": 812,
    "duration": 0
  },
  {
    "name": "first-contentful-paint",
    "entryType": "paint",
    "startTime": 856,
    "duration": 0
  }
]

Solution: Playwright - Performance

Emulation

Test it all, break nothing: Emulation for every situation!

Solution: Playwright - Emulation

Emulalation

// custom.config.js
export {
	viewport: { width: 360, height: 640 },
    devices: [playwright.devices['iPhone 12']],
    geolocation: { latitude: 37.7749, longitude: -122.4194 },
    locale: 'en-US',
    timezone: 'America/Los_Angeles',
    permissions: ['geolocation', 'notifications'],
    colorScheme: 'dark',
    offline: true,
    javascriptEnabled: false
}

// Use it in playwright.config.js
import { defineConfig } from '@playwright/test';
export default defineConfig({ use: { /*...*/ } })
                            
// Use it in test file
import { test } from '@playwright/test'
test.use({ /*...*/ })

Solution: Playwright - Emulation

Multiple configs

import { defineConfig } from '@playwright/test';
export default defineConfig({ projects: [
  {
    name: 'desktop',
    use {
      // ...
    }
  },
  {
    name: 'mobile',
    use {
      // ...
    }
  },
] })
              

Solution: Playwright - Emulation

Chrome DevTool Protocol

Solution: Playwright - Emulation

Chrome DevTool Protocol

import { test } from '@playwright/test'

test('basic performance emulation', async ({ page }) => {
  const cdp = await page.context().newCDPSession(page)
  await cdp.send('Network.enable')
  await cdp.send('Network.emulateNetworkConditions', {
    offline: false,
    downloadThroughput: (4 * 1024 * 1024) / 8,
    uploadThroughput: (3 * 1024 * 1024) / 8,
    latency: 20
  })

  await page.goto('https://playwright.dev/docs')
})

Solution: Playwright - Emulation

Chrome DevTool Protocol

async function measurePerformance(executeSteps, stepName, page) {
      // Add listeners
      page.on('request', request => { /* .. collect some metrics .. */ });
      page.on('response', async response => { /* .. collect some more metrics .. */ } );

      // Execute the steps with performance markers
      const start = performance.now();
      await executeSteps();
      const end = performance.now();

      // Record the performance metrics
      const timings = await page.evaluate(() => performance.timing );
      const navigationTime = await page.evaluate(() => performance.getEntriesByType('navigation') );
     
      perfMetrics[step] = {
         stepName,
         timeElapsed: end - start,
         visualComplete: timings.loadEventEnd - timing.navigationStart,
         navigationTime: navigationTime[0].duration,
         // mooooaaarrr metrics
      };
      step++;   // Step Counter
}

Solution: Playwright - Emulation

Further readings

Documentation

  • Playwright - https://playwright.dev/
  • Playwright emulation - https://playwright.dev/docs/emulation
  • Performance API - https://developer.mozilla.org/en-US/docs/Web/API/Performance_API
  • CDP APIs - https://chromedevtools.github.io/devtools-protocol/
  • Gutenberg Playwright utils - https://developer.wordpress.org/block-editor/reference-guides/packages/packages-e2e-test-utils-playwright/

 

Articles

  • Kinsta - Performance API - https://kinsta.com/blog/performance-api/
  • Checkly - Playwright and Performance APIs - https://www.checklyhq.com/learn/playwright/performance/
  • Performance testing using Playwright - https://medium.com/@sahajk13/performance-testing-using-playwright-ea8f372e76f0

 

Thank you!

Questions ?

linkedin: shady.sharaf

wp.org: shadyvb

github: shadyvb

Playwright

By Shadi Sharaf