UI Automation Testing with Headless Chrome

Kevin C Chen @ 2018 SDET Summit

 Streamlines DevOps
Testing Workflow

https://devopscube.com/what-is-devops-what-does-it-really-mean/

Test features work from the user’s perspective
via Web UI

Challenges

Complex Deployment & Setup
 

Flaky Vs. Robustness
 

Lengthy test runs
 

Cost of upkeeping / tuning with sleep(), wait()

image source: https://cdn.edureka.co/blog/wp-content/uploads/2017/05/Selenium-blog-1.gif

Goals

• Painless to setup, zero configuration, no server
   
• No need to learn another proprietary tool that talks to browser
   
• Instrument browser directly with its native API
   
• Can run both in headless mode in CI/CD pipeline or interactive mode with browser
   
• Access to browser DevTools features

Headless Browser
=
Chrome without Chrome

What is Headless Chrome?

• A way to run the Chrome browser in a headless environment
   
• It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line
   
• A great tool for automated testing and server environments where a visible UI shell is not needed

TL;DR

Why Headless Chrome?

• Stays up to date with the web
   
• Runs fast without actual browser UI
   
• Test all modern features (ES6 / CSS / HTML)
  streams, css grid, push notification, service worker...
   
• Programmatic access to DevTools
  network throttling, device emulation, code coverage...

What can Headless Chrome do?

1. Pre-rendering JS sites / server sider rendering static markup 
2. Validate lazy loading with Code Coverage
3. A/B Testing e.g. Stylesheets Vs Inline critical styles
4. Catch issues for the Google crawler to determine if your app might not render correctly in Google search
5. Create custom PDF
6. Crawl a Single Page Application
7. Verify service worker offline caching
8. Make your browser talk (headful)
9. Test a Chrome extension  (headful)
10. Control keyboard event

https://www.youtube.com/embed/lhZOFUY1weo?enablejsapi=1

Demo Test Run

What is Puppeteer?

• A Node Library open sourced by Google (Chrome DevTools team)
   
• Provides a high-level API to control headless (or full) Chrome or Chromium over the DevTools Protocol
   
• Bundles with latest Chromium: Zero configuration
   
• Canonical reference for the Chrome DevTools protocol
   
• Almost zero performance overhead over an automated page

https://www.youtube.com/embed/lhZOFUY1weo?enablejsapi=1

What can Puppeteer do?

• Most things that you can do manually in the browser!
   
• Built for Automation
   
• JS async/await for asynchronous things to avoid arbitrary wait times
   
• Works for both traditional JSP UI & modern React UI (API based)

https://pptr.dev/#?product=Puppeteer&version=v1.9.0&show=api-overview

How do we perform UI tests?

• Jest - JavaScript Test Framework by Facebook
  • testing structure
  • assertions functions
  • mocks, spies
  • display test result, code coverage reports
     
• Parallelizes test runs across test suites to maximize performance

https://www.youtube.com/embed/lhZOFUY1weo?enablejsapi=1

Test Walkthrough

CI/CD with Docker Container

• Self-contained image that can run on Jenkins CI pipeline
   
• Installation of Chromium can be part of npm dependencies or with a separate download within the container
   
• An on demand regression test run
   
• Run before PR merge with UI changes

Alternatives

• We had Selenium already, why bother?
  • slow, brittle, and costly
  • lot of configuration and unexpected behaviour
     
• But Selenium sometimes makes more sense
  • If our UI has a lot of device or browser specific code, we may still want Selenium
     
• WebDriver and ChromeDriver
   
• Cypress
   
• Trade-offs

Bigger Questions

• How do we test our UI today?
   
• What is our automation coverage for UI test?
   
• A step back: How many UI to test?
   
• What is our test strategy to ensure we deliver a good user experience?

 Testing from the user’s perspective

• Time to interactive (TTI) - how long it takes for app to load and become capable of quickly responding to user interactions

User Perceived Performance?

https://developers.google.com/web/fundamentals/performance/rail

User having to wait for the browser to respond to an event. Amount of pain is directly proportional to the time user waits

• First Input Delay (FID) - measures the delay that users experience when they interact with the page while it's not yet interactive
   

User Perceived Pain?

https://developers.google.com/web/updates/2018/05/first-input-delay

RAIL - User-centric Performance Model

https://developers.google.com/web/fundamentals/performance/rail

produce a frame < 10ms

process events
< 50ms

maximize

Time To Interactive
< 5s

How to Measure: Lighthouse

• Headless Browser
• How to automate with Puppeteer
• How to build UI automation test suite with Jest
• CI/CD integration with Docker container
• Alternative approaches
   
• UI testing strategy and considerations

What We've Learnt

References

• Puppeteer
  • https://pptr.dev/
  • https://github.com/GoogleChrome/puppeteer
• Jest
  • https://jestjs.io/en/
• Diagrams based on this repo