About Me

Web UI Test Engineer,

Space Mission Control Software, KBR

NASA Ames Research Center

Open MCT

VIPER

In this talk

Clientside Performance Testing
What to Measure
Playwright
Playwright Capabilities
Playwright Problems (and Solutions)
State of the Union and What's Next!

github.com/unlikelyzero/js-perf-toolkit

In this talk

How to fix performance problems.
Part 1!

What is Clientside Performance?

"User Experience... using a web application... in a web browser"
Easiest to contextualize when things go wrong

Happy Accidents

What to measure?

After the first load
Network
- Still matters after the first load!
- Latency
- Number of requests
CPU / GPU
- "Jank" or "Freezing"
- Frames Per Second
Memory
- Leaks.

Where do measurements come from?

Chrome, itself
Browser/Web Standard APIs
DevTools and DevTools APIs

Browser/Web APIs

Specification on MDN
ResourceTiming APIs
- How long does an individual resource request take and where does it spend it's time
LongTasks APIs
- Provides a great proxy for CPU/GPU/Animation
- Keeps track of where the main thread is locked for 50ms or more
UserTiming API
- Performance.mark()
- Twitter time-to-first-tweet example

Google Search Devtools Demo

DevTools - Performance Tab

How do we get to these Chrome and Web measurements.... programmatically and automatically?

Based on puppeteer project or "how Chrome tests Chrome"
Supports Edge, Chrome, Firefox, Safari, iOS(ish)
Java, CSharp, Python, Python Asyncio, JS, Typescript
Tighter coupling to browser internals than Selenium
aka "Inside-out Performance Measurements"

Getting started with Playwright Demo

 npx install playwright
npx playwright codegen

github.com/unlikelyzero/js-perf-toolkit

Playwright Capabilities Demo

Already showed how easy it is to get started
Let's demonstrate the power
Capabilities Demo 1: CDP and Performance API
Capabilities Demo 2: Generated Performance.mark() and DevTool Performance Timeline.
Capabilities Demo 3: API Interception and FPS Measurement

WIP Capabilities

These "work" but are not proven
PerformanceObserver, LongTasks
HAR file generated during test.
Send HAR to k6 or Jmeter to simulate many users

Challenges (and solutions) in Playwright Performance Testing

"Test Drift"

Synthetic tests aren’t very resilient, and they can easily fail when small UI changes are introduced, generating unnecessary alert noise. This means that whenever a minor application element such as a button is changed, the corresponding test must be, too."

dynatrace.com/news/blog/what-is-synthetic-monitoring/

https://github.com/nasa/openmct/pull/5203/files

Network Fix - Remove it

Your Internal Network OR
Localhost

CDP

Synthetic User Monitor

"Internet"

Network Fix - Control it

Inside the browser with CDP access
Inject CDP Network Profiles

 const client = await page.target().createCDPSession();
await client.send('Network.enable');
await client.send('Network.emulateNetworkConditions', {
  latency: 500 
});

Network Control Pt 2

Once you're in the browser, you control it all
Mock Network Responses with API Interception
Modify Request OR Response on the wire

 await page.route('**/*.png', route => {
  route.fulfill({
    body: './bing.png'
  });
});

Measurement "Transferability"

Named Performance Machines?
Your machine is not the same as mine.

"Chrome as a service" and way more than today's demo
"Interface for Performance Testing"
Chrome + essentials in a container. Hard.
Exposes generic CDP endpoint
Playwright remotely connects via .connectOverCPD() API
Transferability!

browserless

 browserless:
    image: browserless/chrome:latest
    cpus: 2.0
    mem_limit: 4g

github.com/browserless/chrome

Remote CDP

Your Internal Network

connectOverCDP()

emulateNetworkConditions

0 latency

500ms

"3g"

Machine Transferability

Create a Machine Baseline to store alongside results

 test('Zeroed baseline', async ({page, browser}) => {
  await console.time('htmlfloor');
  await page.setContent(`
    <div class="visible">Hello world</div>
    <input type="checkbox" enabled class="enabled">
  `);
  await page.locator('.visible').isVisible();
  await console.timeLog('htmlfloor');
}); test('Zeroed baseline', async ({page, browser}) => {
  await console.time('htmlfloor');
  await page.setContent(`
    <div class="visible">Hello world</div>
    <input type="checkbox" enabled class="enabled">
  `);
  await page.locator('.visible').isVisible();
  await console.timeLog('htmlfloor');
}); test('Zeroed baseline', async ({page, browser}) => {
  await console.time('htmlfloor');
  await page.setContent(`
    <div class="visible">Hello world</div>
    <input type="checkbox" enabled class="enabled">
  `);
  await page.locator('.visible').isVisible();
  await console.timeLog('htmlfloor');
});

Precision, Accuracy, and Variability

Leverage "Browser Time" from Performance APIs. This includes measurements in the microsecond!
Playwright has performance changes
Chrome has performance changes
Run over run variation
How do we know if a 10% decrease in a metric in is an application change or... run-over-run variation?
Controlled as many variables as possible!
Run a lot. a lot a lot.
Becomes a data problem.

InfluxDB JS Client

 const { influx } = require('./influxUtils');
 
test.describe('feature foo', () => {
  test('my test', async ({ page }) => {
    await page.goto('/');
    await influx.write(
      'test_mesurement',
      { tag1: 'tag 1 value' },
      { field1: 12, field2: 2.3 },
      Date.now()
    );
  });
});

Solve the "Data" problem and push results into TSDB
InfluxDB has a native InfluxDB Client!
InfluxDB Allows you to modify timestamps vs Prometheus Write's limitation to use TSDB-received time

Summary

Clientside Performance
What to Measure
Playwright
Automated(?) a Google Search Performance Test
Playwright Performance Testing problems and Solutions

github.com/unlikelyzero/js-perf-toolkit

What's next?

xk6-browser

Open Source
k6 driving playwright APIs
beta
supports .connectOverCDP()

What it solves:

*Generate many, many
measurements!

** Tap into the huge k6 ecosystem to integrate and correlate with System Monitoring, Load Testing, etc

My next steps

Push everything to Performance Observer
Automated Memory Leak detection
Performance Timeline Parsing Library?

Help NASA Open MCT!

Github Discussion

Open e2e Testing Initiative
Your opportunity to contribute to NASA
Traditional CLA applies to testers
It's as easy as it looked!

Contact and Links

openmct

js-perf-toolkit

browserless/chrome

xk6-browser

Why is frontend performance important?

Amazon found that every 100 milliseconds in load time correlated to 1% decrease in sales.
Google found that when they increased the search results from 10 to 30, the load time increased by half a second and resulted in a 20% decrease in ad revenues.
Frontend is almost always the bearer of bad news.

	performance.mark('first-google-ad');
	### Some number of clicks or steps ###
	performance.mark('click-on-first-ad');
	performance.measure('first-ad-click-duration','first-google-ad','click-on-first-ad');

	###returned object
	PerformanceMeasure {
	name: 'first-ad-click-duration',
	entryType: 'measure',
	startTime: 4727964.299999952,
	duration: 12436.700000047684
	}

	performance.mark('first-google-ad');
	### Some number of clicks or steps ###
	performance.mark('click-on-first-ad');
	performance.measure('first-ad-click-duration','first-google-ad','click-on-first-ad');

	###returned object
	PerformanceMeasure {
	name: 'first-ad-click-duration',
	entryType: 'measure',
	startTime: 4727964.299999952,
	duration: 12436.700000047684
	}

	performance.mark('first-google-ad');
	### Some number of clicks or steps ###
	performance.mark('click-on-first-ad');
	performance.measure('first-ad-click-duration','first-google-ad','click-on-first-ad');

	###returned object
	PerformanceMeasure {
	name: 'first-ad-click-duration',
	entryType: 'measure',
	startTime: 4727964.299999952,
	duration: 12436.700000047684
	}

	performance.mark('first-google-ad');
	### Some number of clicks or steps ###
	performance.mark('click-on-first-ad');
	performance.measure('first-ad-click-duration','first-google-ad','click-on-first-ad');

	###returned object
	PerformanceMeasure {
	name: 'first-ad-click-duration',
	entryType: 'measure',
	startTime: 4727964.299999952,
	duration: 12436.700000047684
	}

	performance.mark('first-google-ad');
	### Some number of clicks or steps ###
	performance.mark('click-on-first-ad');
	performance.measure('first-ad-click-duration','first-google-ad','click-on-first-ad');

	###returned object
	PerformanceMeasure {
	name: 'first-ad-click-duration',
	entryType: 'measure',
	startTime: 4727964.299999952,
	duration: 12436.700000047684
	}

	const client = await page.target().createCDPSession();
	await client.send('Network.enable');
	await client.send('Network.emulateNetworkConditions', {
	latency: 500
	});

	await page.route('*/.png', route => {
	route.fulfill({
	body: './bing.png'
	});
	});

	browserless:
	image: browserless/chrome:latest
	cpus: 2.0
	mem_limit: 4g

	test('Zeroed baseline', async ({page, browser}) => {
	await console.time('htmlfloor');
	await page.setContent(`
	<div class="visible">Hello world</div>
	<input type="checkbox" enabled class="enabled">
	`);
	await page.locator('.visible').isVisible();
	await console.timeLog('htmlfloor');
	});

	const { influx } = require('./influxUtils');

	test.describe('feature foo', () => {
	test('my test', async ({ page }) => {
	await page.goto('/');
	await influx.write(
	'test_mesurement',
	{ tag1: 'tag 1 value' },
	{ field1: 12, field2: 2.3 },
	Date.now()
	);
	});
	});

Performance Testing with Playwright