End-to-end testing with Playwright

Sixian Li, 2023/02

Agenda

  • Why E2E testing
  • Playwright vs. Cypress
  • Write, run, debug tests in Playwright
  • What about TAB?

Why E2E

Unit tests are fast and easy to run, but they can’t provide a complete picture of the system's functionality.

Why E2E

Pros

  1. A realistic view of user scenarios
  2. Uncover issues that might not be detected by unit tests
  3. Confidence that the system works as intended for the end-user

Why E2E

Pros

  1. A realistic view of user scenarios
  2. Uncover issues that might not be detected by unit tests
  3. Confidence that the system works as intended for the end-user

Cons

  1. Slower to run and more complex to set up
  2. Need more effort to maintain. They are sensitive to changes in the UI and environment, so need to be updated more frequently.

Playwright vs. Cypress

Playwright vs. Cypress

Playwright
Straightforward. Just enter the credentials.

test('test', async ({ page }) => {
  await page.goto('https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH');
  await page.getByPlaceholder('Email, phone, or Skype').fill('admin@a830edad9050849tesliomsit.onmicrosoft.com');
  await page.getByPlaceholder('Email, phone, or Skype').press('Enter');
  await page.getByPlaceholder('Password').press('Enter');
  await page.getByPlaceholder('Password').fill(password);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.getByRole('button', { name: 'Yes' }).click();
});

Authentication

Playwright vs. Cypress

Authentication

Cypress
No easy way. Need additional sign in script and library(node-sp-auth). Cross-origin tests are just supported in Cypress 12 released in Dec. 2022, 9 years after its first release.

Playwright vs. Cypress

Authentication

Cypress is in-process web automation. It injects itself into the web page, and drives tests using Web API. 

https://youtu.be/PXTspGn1im0

Playwright vs. Cypress

Authentication

Playwright is out-of-process. It connects to the browser using Chrome Devtool Protocol.

https://youtu.be/PXTspGn1im0?t=249

Playwright vs. Cypress

Playwright
Parallel and headless by default, no config at all.
> All tests run in worker processes. These processes are OS processes, running independently, orchestrated by the test runner. All workers have identical environments and each starts its own browser.

Speed

Cypress
Need additional configuration. They have a long page about parallelization: https://docs.cypress.io/guides/guides/parallelization

Playwright vs. Cypress

Playwright
Chromium, Firefox, WebKit

Supported browsers

Cypress
WebKit is still experimental. Cross-origin issue is not resolved yet.

Playwright vs. Cypress

Playwright
 

Selector

Playwright
Same as testing-library. Our UT knowledge is transferrable.
 

 

await expect(page.getByRole('heading', { name: 'Sign up' })).toBeVisible();

await page.getByRole('checkbox', { name: 'Subscribe' }).check();

await page.getByRole('button', { name: /submit/i }).click();

Playwright vs. Cypress

Playwright
 

Selector

Cypress
CSS selector by default. Need to install testing-libnrary separately.

Playwright
Same as testing-library. Our UT knowledge is transferrable.
 

 

cy.get('input').should('be.disabled')

cy.get('ul li:first').should('have.class', 'active')

Playwright tests

Write

playwright codegen https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

Playwright tests

Write

playwright codegen https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

Playwright tests

Write

async function addExternalLink(page: Page, topicPage: TopicPage, link: string, linkName: string) {
  await topicPage.edit();
  await page.getByRole('button', { name: 'Add a file or page for this topic' }).click();

  await page.locator('button[role="presentation"]:has-text("From a link")').click();

  await page.getByPlaceholder('https://').fill(link);
  await page.getByRole('textbox', { name: 'Name *' }).fill(linkName);
  await page.getByRole('button', { name: 'Add' }).click();

  await topicPage.publish();

  expect(page.getByRole('link', { name: linkName })).toHaveCount(1);
}

Playwright tests

Write - Preserve authenticated state

playwright codegen --save-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

playwright codegen --load-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

test.use({
  storageState: 'auth.json'
});
playwright codegen --save-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

playwright codegen --load-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH

test.use({
  storageState: 'auth.json'
});

Playwright tests

Write - Reuse signed in state

// global-setup.ts
import { chromium, FullConfig } from '@playwright/test';

async function globalSetup(config: FullConfig) {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://github.com/login');
  await page.getByLabel('Username or email address').fill('username');
  await page.getByLabel('Password').fill('password');
  await page.getByRole('button', { name: 'Sign in' }).click();
  // Save signed-in state to 'storageState.json'.
  await page.context().storageState({ path: 'storageState.json' });
  await browser.close();
}

export default globalSetup;

Playwright tests

Write - Page Object Model

export class TopicPage {
  readonly page: Page;
  constructor(page: Page) {
    this.page = page;
  }

  async create() {
	...
  }

  async delete() {
    await this.page.getByRole('menuitem', { name: 'Page details' }).click();
    await this.page.getByRole('button', { name: 'Delete page' }).click();
    await this.page.getByRole('button', { name: 'Delete' }).click();
    await this.page.waitForURL(`${KNOWLEDGE_HUB_URL}/SitePages/Home.aspx`);
  }

  async edit() {
   ...
  }

  async publish() {
	...
  }
}

Playwright tests

Write - Page Object Model

export class ManageTopicsPage {
  readonly page: Page;
  constructor(page: Page) {
    this.page = page;
  }

  async goToRemovedView() {
    await this.page.getByRole('button', { name: 'View removed topics' }).click();
    await expect(this.page.getByRole('heading', { name: 'Removed topics you can manage' })).toBeVisible();
    await expect(
      this.page.getByRole('button', { name: 'Trend of topics by status over the past 30 days' })
    ).toBeHidden();
  }
  
  async returnToActiveView() {
	...
  }

  async openFilterPane() {
    ...
  }
}

Playwright tests

Write - Fixture

export const test = base.extend<Fixtures>({
  topicPage: async ({ page }, use) => {
    const topicPage = new TopicPage(page);
    await topicPage.create();
    await use(topicPage);
    await topicPage.delete();
  },
  manageTopicsPage: async ({ page }, use) => {
    const manageTopicsPage = new ManageTopicsPage(page);
    await manageTopicsPage.load();
    await use(manageTopicsPage);
  }
});

Playwright tests

Write - Use fixture

 test('should allow adding external resource from link tab', async ({ page, topicPage }) => {
   
   await topicPage.edit();
   await page.getByRole('button', { name: 'Add a file or page for this topic' }).click();

   await page.locator('button[role="presentation"]:has-text("From a link")').click();

   await page.getByPlaceholder('https://').fill(link);
   await page.getByRole('textbox', { name: 'Name *' }).fill(linkName);
   await page.getByRole('button', { name: 'Add' }).click();

   await topicPage.publish();

   expect(page.getByRole('link', { name: linkName })).toHaveCount(1);
   
 });

Playwright tests

Write - Snapshot

expect(await page.screenshot()).toMatchSnapshot();
await expect(page.getByRole('link', { name: newName })).toHaveCount(1);

await hideElements([page.getByRole('button', { name: /feedback/i })]);

expect(
  await topicPage.featureTagLocator('ResourcesWebPart').screenshot({ animations: 'disabled' })
).toMatchSnapshot();

Playwright tests

Run

// Run all tests
playwright test

// Run test with specific title
playwright test -g "add a todo item"

// Headed mdoe
playwright test landing-page.spec.ts --headed

// Specific project
playwright test landing-page.ts --project=chromium

Playwright tests

Run - Report

Playwright tests

Run - Report

Playwright tests

Debug

Playwright tests

Debug - VSCode extension

What about TAB test?

Hand written execution log

Find elements by automation id

Slow pipeline

Can't even import code from source file

Thank you