End-to-end testing with Playwright
Sixian Li, 2023/02
Agenda
- Why E2E testing
- Playwright vs. Cypress
- Write, run, debug tests in Playwright
- What about TAB?
Why E2E
Unit tests are fast and easy to run, but they can’t provide a complete picture of the system's functionality.
Why E2E
Pros
- A realistic view of user scenarios
- Uncover issues that might not be detected by unit tests
- Confidence that the system works as intended for the end-user
Why E2E
Pros
- A realistic view of user scenarios
- Uncover issues that might not be detected by unit tests
- Confidence that the system works as intended for the end-user
Cons
- Slower to run and more complex to set up
- Need more effort to maintain. They are sensitive to changes in the UI and environment, so need to be updated more frequently.
Playwright vs. Cypress
Playwright vs. Cypress
Playwright
Straightforward. Just enter the credentials.
test('test', async ({ page }) => {
await page.goto('https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH');
await page.getByPlaceholder('Email, phone, or Skype').fill('admin@a830edad9050849tesliomsit.onmicrosoft.com');
await page.getByPlaceholder('Email, phone, or Skype').press('Enter');
await page.getByPlaceholder('Password').press('Enter');
await page.getByPlaceholder('Password').fill(password);
await page.getByRole('button', { name: 'Sign in' }).click();
await page.getByRole('button', { name: 'Yes' }).click();
});Authentication
Playwright vs. Cypress
Authentication
Cypress
No easy way. Need additional sign in script and library(node-sp-auth). Cross-origin tests are just supported in Cypress 12 released in Dec. 2022, 9 years after its first release.
Playwright vs. Cypress
Authentication
Cypress is in-process web automation. It injects itself into the web page, and drives tests using Web API.
Playwright vs. Cypress
Authentication
Playwright is out-of-process. It connects to the browser using Chrome Devtool Protocol.
Playwright vs. Cypress
Playwright
Parallel and headless by default, no config at all.
> All tests run in worker processes. These processes are OS processes, running independently, orchestrated by the test runner. All workers have identical environments and each starts its own browser.
Speed
Cypress
Need additional configuration. They have a long page about parallelization: https://docs.cypress.io/guides/guides/parallelization
Playwright vs. Cypress
Playwright
Chromium, Firefox, WebKit
Supported browsers
Cypress
WebKit is still experimental. Cross-origin issue is not resolved yet.
Playwright vs. Cypress
Playwright
Selector
Playwright
Same as testing-library. Our UT knowledge is transferrable.
await expect(page.getByRole('heading', { name: 'Sign up' })).toBeVisible();
await page.getByRole('checkbox', { name: 'Subscribe' }).check();
await page.getByRole('button', { name: /submit/i }).click();Playwright vs. Cypress
Playwright
Selector
Cypress
CSS selector by default. Need to install testing-libnrary separately.
Playwright
Same as testing-library. Our UT knowledge is transferrable.
cy.get('input').should('be.disabled')
cy.get('ul li:first').should('have.class', 'active')Playwright tests
Write
playwright codegen https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
Playwright tests
Write
playwright codegen https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
Playwright tests
Write
async function addExternalLink(page: Page, topicPage: TopicPage, link: string, linkName: string) {
await topicPage.edit();
await page.getByRole('button', { name: 'Add a file or page for this topic' }).click();
await page.locator('button[role="presentation"]:has-text("From a link")').click();
await page.getByPlaceholder('https://').fill(link);
await page.getByRole('textbox', { name: 'Name *' }).fill(linkName);
await page.getByRole('button', { name: 'Add' }).click();
await topicPage.publish();
expect(page.getByRole('link', { name: linkName })).toHaveCount(1);
}Playwright tests
Write - Preserve authenticated state
playwright codegen --save-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
playwright codegen --load-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
test.use({
storageState: 'auth.json'
});playwright codegen --save-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
playwright codegen --load-storage=auth.json https://a830edad9050849tesliomsit.sharepoint.com/sites/KnowledgeH
test.use({
storageState: 'auth.json'
});Playwright tests
Write - Reuse signed in state
// global-setup.ts
import { chromium, FullConfig } from '@playwright/test';
async function globalSetup(config: FullConfig) {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://github.com/login');
await page.getByLabel('Username or email address').fill('username');
await page.getByLabel('Password').fill('password');
await page.getByRole('button', { name: 'Sign in' }).click();
// Save signed-in state to 'storageState.json'.
await page.context().storageState({ path: 'storageState.json' });
await browser.close();
}
export default globalSetup;Playwright tests
Write - Page Object Model
export class TopicPage {
readonly page: Page;
constructor(page: Page) {
this.page = page;
}
async create() {
...
}
async delete() {
await this.page.getByRole('menuitem', { name: 'Page details' }).click();
await this.page.getByRole('button', { name: 'Delete page' }).click();
await this.page.getByRole('button', { name: 'Delete' }).click();
await this.page.waitForURL(`${KNOWLEDGE_HUB_URL}/SitePages/Home.aspx`);
}
async edit() {
...
}
async publish() {
...
}
}
Playwright tests
Write - Page Object Model
export class ManageTopicsPage {
readonly page: Page;
constructor(page: Page) {
this.page = page;
}
async goToRemovedView() {
await this.page.getByRole('button', { name: 'View removed topics' }).click();
await expect(this.page.getByRole('heading', { name: 'Removed topics you can manage' })).toBeVisible();
await expect(
this.page.getByRole('button', { name: 'Trend of topics by status over the past 30 days' })
).toBeHidden();
}
async returnToActiveView() {
...
}
async openFilterPane() {
...
}
}
Playwright tests
Write - Fixture
export const test = base.extend<Fixtures>({
topicPage: async ({ page }, use) => {
const topicPage = new TopicPage(page);
await topicPage.create();
await use(topicPage);
await topicPage.delete();
},
manageTopicsPage: async ({ page }, use) => {
const manageTopicsPage = new ManageTopicsPage(page);
await manageTopicsPage.load();
await use(manageTopicsPage);
}
});Playwright tests
Write - Use fixture
test('should allow adding external resource from link tab', async ({ page, topicPage }) => {
await topicPage.edit();
await page.getByRole('button', { name: 'Add a file or page for this topic' }).click();
await page.locator('button[role="presentation"]:has-text("From a link")').click();
await page.getByPlaceholder('https://').fill(link);
await page.getByRole('textbox', { name: 'Name *' }).fill(linkName);
await page.getByRole('button', { name: 'Add' }).click();
await topicPage.publish();
expect(page.getByRole('link', { name: linkName })).toHaveCount(1);
});Playwright tests
Write - Snapshot
expect(await page.screenshot()).toMatchSnapshot();await expect(page.getByRole('link', { name: newName })).toHaveCount(1);
await hideElements([page.getByRole('button', { name: /feedback/i })]);
expect(
await topicPage.featureTagLocator('ResourcesWebPart').screenshot({ animations: 'disabled' })
).toMatchSnapshot();
Playwright tests
Run
// Run all tests
playwright test
// Run test with specific title
playwright test -g "add a todo item"
// Headed mdoe
playwright test landing-page.spec.ts --headed
// Specific project
playwright test landing-page.ts --project=chromiumPlaywright tests
Run - Report
Playwright tests
Run - Report
Playwright tests
Debug
Playwright tests
Debug - VSCode extension
What about TAB test?
Hand written execution log
Find elements by automation id
Slow pipeline
Can't even import code from source file