About Me

Web UI Test Engineer, KBR

NASA Ames Research Center

What is Open MCT?

github.com/nasa/openmct

Red Bull Formula One

VIPER

In this talk

  • Performance Modeling
  • Performance in Open MCT Context
  • openmct-performance
  • Challenges in FE Performance

Automated Performance Modeling

I didn't go to Art School

Users

Backend

API

OS

Web App

Frontend

Open MCT

YAMCS

CouchDB

k8s

Docker

Bearer of Bad News

BE Issues = FE Issues

Users

Backend

API

OS

Web App

Frontend

Happy Accidents

Frontend Changes = Backend Problems

  • Google found that when they increased the search results from 10 to 30, the load time increased by half a second and resulted in a 20% decrease in ad revenues.
  • Intentional Product Change
  • One Change. Everyone's problem.

PDM Changes = BE Problems

Users

Backend

API

OS

Web App

Frontend

UX Performance

  • Twitter "Time to First Tweet"
  • User Timing API
  • Performance.marks() API
performance.mark('search-entered');
### Some number of clicks or steps ###
performance.mark('search-returned');
performance.measure('total-search-time','search-entered,'search-returned');

###returned object
PerformanceMeasure {
  name: 'total-search-time',
  entryType: 'measure', 
  startTime: 4727964.299999952, 
  duration: 12436.700000047684
}

Open MCT Search Demo

 

  • openmct-quickstart
  • Universal Search
  • "Guess the traffic"
  • HAR File

github.com/scottbell/openmct-quickstart

Open MCT Performance

  • How do we prevent Open MCT from just being the bearer of bad news?
  • openmct-performance is an "example repo" on how to write a performance test with Open MCT
  • Companion to openmct-quickstart
  • 'docker compose up'***

github.com/unlikelyzero/openmct-performance

  • k6 is an open source load testing tool with a huge ecosystem of extensions and protocol-specific testing capabilities
  • k6 Browser drives real chrome browsers with playwright APIs
  • JavaScript - the lingua franca of the FE
  • Measurements feed into many (all?) storage backends

Demo Time

  • openmct-performance and k6
  • Record a browser test with playwright tooling
  • Record and convert HARfiles with har-to-k6
  • Modeling (combine Browser + HTTP + APM + Infra)

Silos & Asterisks

  • FE devs silo'd from the performance testing and infra teams
  • Performance test tooling can't run in prod/QA/Staging
  • Synthetic User Monitoring is fragile and owned by the wrong team

Synthetic tests aren’t very resilient, and they can easily fail when small UI changes are introduced, generating unnecessary alert noise. This means that whenever a minor application element such as a button is changed, the corresponding test must be, too."

dynatrace.com/news/blog/what-is-synthetic-monitoring/

Performance Test Results

from browserstack.com/speedlab

  • Baseline Function

Keep k6-browser tests in sync with development

  • UI Locator Contract Tests
    Playwright Locator Changes ->
    openmct PRs -> openmct-performance


     
### Automated e2e test run on every PR
await page.locator('[aria-label="OpenMCT Search"] input[type="search"]').click();

### Automated k6 test run downstream
await page.locator('[aria-label="OpenMCT Search"] input[type="search"]').click();
  • Community Shoutout!
    • Playwright Functional + k6-browser shared repo!
    • https://github.com/ticup/xk6-browser-template-typescript-playwright

Summary

  • Performance Modeling
  • Open MCT Performance
  • Integrated Frontend and Backend Performance
  • How to Keep Frontend Performance In Sync With Development

Thanks!
 

Contact and Links

openmct

openmct-performance

k6-browser

Cutting Room Floor

Synthetic User Monitoring

  • Recent Practice of Automated Performance Testing 
  • "SRE Team running Selenium tests"
  • Why? The frontend is bearer of bad news!
  • Critical User Journeys covered
  • Comes with a lot of asterisks
First Load Loaded Page Session

Network Fix - Control it

  • Inside the browser with CDP access
  • Inject CDP Network Profiles
const client = await page.target().createCDPSession();
await client.send('Network.enable');
await client.send('Network.emulateNetworkConditions', {
  # 5% Decrease in Ad revenue
  latency: 500 
});

Network Control Pt 2

  • Once you're in the browser, you control it all
  • Mock Network Responses with API Interception
  • Note: Couldn't Replace 10 -> 30 results :(
await page.route('**/*.png', route => {
  route.fulfill({
    body: './bing.png'
  });
});

Synthetic User Monitoring Challenges (Part Deux)

And lastly, many synthetic monitoring tools lack the context needed to explain why a specific failure happened or what the business implications might be, lengthening time to resolution and making it unnecessarily difficult to prioritize application performance issues."

dynatrace.com/news/blog/what-is-synthetic-monitoring/

***** Asterisk

xk6-browser

  • Open Source
  • k6 driving playwright APIs
  • beta
  • supports .connectOverCDP()

What it solves:

*Generate many, many
measurements!

** Tap into the huge k6 ecosystem to integrate and correlate with System Monitoring, Load Testing, etc

 

Creating a baseline of measurement for transferability

  • Let’s get rid of all asterisks associated with our measurment.
  • Create a HTML Webpage.
  • Test and Time the page.
  • Demo
  • Run this before every run. Run between version changes. Run this local vs CI.

Summary

  • Browser Performance and DevTools
  • Load Testing and Perf Problems
  • What to Measure
  • Browserless
  • Driving with Puppeteer
  • js-perf-toolkit

Demo

What do we need?

  • Stable Interface
  • Controlled Environment

Demo

What do we need?

  • Stable Interface
  • Playwright "bless your own damn build" testing
  • sitespeed.io is a better specialized tool

Hard?

  • Variability
    • We "only" want App-under-test variation
    • Network
    • CPU
    • Chrome
    • Test Framework
  • Transferability
    • CPU/GPU
  • QA Team giving us bad builds!

"Hard" Demo

Capabilities Demo

JS-Perf-Toolkit

  • github.com/unlikelyzero/js-perf-toolkit
  • Moves everything* into containers
    • browserless
    • influxdb+prometheus+grafana
    • Puppeteer/Playwright Examples
    • NodeJS->InfluxDB (coming soon)
    • xk6-browser*
  • Network control with toxiproxy
  • Integrates into monitoring systems with InfluxDB and Prometheus*

Why?

  • Load Testing is what you do to the backend to approximate Rough Response Time
  • HTTP Rest API can provide a rough estimate to User Experiences in the app

Selenium?

  • Yes? No? Maybe?
  • Variability in results due to waitFor
  • Driver? Jmeter+Selenium
  • 2 Cores per User!
  • CDP in 4.0

How?

 

(Chrome! Devtools!)

Our first asterisk * !

Load Testing Can't Properly Approximate Frontend Performance Regressions

Chrome

  • Loads and renders your web app via static assets, javascript, and APIs
  • Web app / javascript
    • How well it runs on your machine
  • Static Assets
    • You can't
  • But!
    • HTTP 1.1 only allows 6 sequential API responses
    • Websockets?
    • Graphql queries based on user data?

Network Variability*

  • Network is defined by Latency and number of requests necessary to provide User the needed data to present on screen
  • Variability in the internet and local machine

 

Why is *frontend* performance important?

  • Amazon found that every 100 milliseconds in load time correlated to 1% decrease in sales.
  • Google found that when they increased the search results from 10 to 30, the load time increased by half a second and resulted in a 20% decrease in ad revenues.
  • Load time and User Experience

Quick Check-in

  • Why performance test
    • Load times and lighthouse
  • What to look for
    • After load and Long Tasks
  • RUM, User Timing, and marks

How did Google know the users left?

Google found that when they increased the search results from 10 to 30, the load time increased by half a second and resulted in a 20% decrease in ad revenues."

Real User Monitoring (RUM)

  • "Where real users go and what they do"
  • Inject js code into application to report back your marketing team
  • Session Replay
  • Key Takeaway for Performance Testing
    • User Timing
    • Performance.marks()

What happens after the first load?

What happens after the first load?

k6

By John Hill

k6

Front-end performance testing is hard. Really hard. There are hundreds of variables that affect end users’ perceived performance. Only a few are measured with traditional load testing tools. Few can be actively controlled outside of a dedicated test environment, and we lose credibility as soon as our tests leave that environment. Worse yet, the available front-end performance tooling blindly focuses on how quickly a page loads. What happens after the first load? At NASA Ames, our Mission Operators have an 18-hour shift. Then there is automation, none of those tools were designed to be automated like our e2e tests. Adding all this up means that, as performance testers, we’re required to add too many asterisks to our results… until now! In this live demo, we’ll detail the metrics that matter and how to measure them without asterisks. Using Browserless, Playwright, and k6, we’ll instrument and automate a performance test. We’ll have a front-end performance tooling state-of-the-union to outline what’s happening in this space and where we’re going. Lastly, we’ll end with ways to integrate these new tools into your existing CI/CD process and test frameworks.

  • 229

More from John Hill