Life after Selenium

John Hill

About Me

Web UI Test Engineer,

Space Mission Control Software, KBR

NASA Ames Research Center

unlikelyzero.com

Open MCT

https://github.com/nasa/openmct

https://openmct-demo.herokuapp.com/

VIPER Mission

Open e2e Testing Initiative

Github Discussion Link

In this talk

  • Selenium
  • Problems
  • Solutions
  • Walk through a test in 3 frameworks
  • What happens next
  • Q&A

github.com/unlikelyzero/life-after-selenium

In this talk

  • Convince you to delete and restart
  • Get commission
  • Mobile
  • All alternatives
  • python, java

In the beginning... 

QTP(now UTP)

  • Created By Mercury Interactive in 1998
  • First(?) Browser UI Automation tool
  • Closed source. Proprietary.
  • $3995 per copy (in 1998)
  • Limited Browser support
  • Windows Only

Selenium

  • "Selenium is the cure for Mercury Poisoning"
  • Created in 2004.
  • Open Source, Open Standard
  • Designed to be cross browser
  • Now maintained by SeleniumHQ.
  • Selenium (2021) = the infra + tooling to run webdriver
  • Webdriver (2021) = part of the w3c browser spec
  • Now cross browser by definition!

Selenium Diagram

Demo Cypress Real World App

Let's Test!

  1. Launch Browser
  2. Login
  3. Click 'New Transaction' Button
  4. Search for 'Devon Becker'
  5. Verify 'Devon Becker' appears

Problematic Selenium Demo

Let's "Fix" it!

Still no XHR :(

WTF (Why the flake)

  • NOT Event-driven
  • Polling for state vs Listening for events
  • Open Standard = "Loose coupling"
  • Lack of observability

Speed / Performance

  • REST HTTP is heavily impacted by network latency
  • Comcast Chaos Engineering
  • Test: 100 ms of Network Latency

Insert 100ms of "comcast" here

Same Test with 100ms of "comcast"

Is our "comcast" test unrealistic?

ping ondemand.saucelabs.com

Mo' Problems

Lots to Manage

  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version

While preparing for today

HW Resources

  • Expensive to run
  • Jmeter example 1000:1 Ratio
  • Video Recording via VNC extremely expensive
    (+1 Core, +1 GB of Memory, Storage)

Selenium as a Service

Problem Summary

  1. Open.
    • Lots of dependencies without ownership
  2. Abstraction.
    • Speed / Performance Issues
    • Flaky due to "loose coupling"
  3. Expensive to run

What's happened since 2004?

i.e.

"Solutions"

Puppeteer

  • "How Chrome tests Chrome"
  • Chrome DevTools Protocol (CDP) API
    • API matches webdriver
  • Ownership
    • Evergreen
    • NodeJS client language binding
    • Chromium binary
    • Cheap and lightweight
  • Firefox support* in Nightlies
  • Not a Framework. Just a Selenium/Webdriver replacement.

Puppeteer (continued)

  • Event-driven
  • Websocket interface to chrome browser
    • Minimal Impact from Network Latency
  • CDP exposes Native Browser Interaction
    • Intercept and interact with network calls

Puppeteer vs Selenium

Quick Summary

Webdriver CDP
w3c standard Yes No
Interaction Polling Real-time
Dependency Heck Heaven
Cross-Browser Yes Sorta*
Inherently Flaky Yes No
"Weight" Heavy Light

Webdriver.io

  • Open Source
  • Originally Webdriver-based.
  • Auto-wait, auto-retry
  • Managed Deps
  • Best example of CDP's impact.
  • Puppeteer is now default!

Demo

  • "Cross-Browser testing done right"
  • Open Source*
  • CDP API shim
  • Controversial
    • MS Hired Puppeteer team
    • Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
  • Can run WebKit (the upstream safari browser engine) on Windows and linux!
  • Now a framework with @playwright/test!
    Easy to migrate to/from puppeteer.
  • Puppeteer + Actionability API

 

 

 

Demo

  • Proprietary Proxy Server. Not CDP.
  • Freemium OSS*
  • More than a framework
    • All in One solution
    • Documentation!
    • GUI Playground
    • *Dashboard service
  • Cross-platform**
    • Firefox
    • Edge

Demo

Where are we now?

Solutions Summary

  • Realtime without Flake!
  • Fewer dependencies and maintenance
  • Cross Browser!
  • Open Source*

F U T U R E

CDP "Standards"

QTP Slide (Revisited)

  • Closed source. Proprietary.
  • $3995 per copy (in 1998)
  • Limited Browser support
  • Windows Only

4.0

Selenium

Summary

  • Selenium
  • Problems
  • Solutions
  • Walked through 3 frameworks
  • What happens next

Q&A

(Contact info)

github.com/unlikelyzero

unlikelyzero.com
ministryoftesting.slack.com

Cutting room floor

Docker-selenium solves
some Network and Dependencies

What is a web browser?

  • Serves up web applications
  • Rendering Engine for Javascript (i.e. Chrome = Blink)
  • Browser receives Webapp code via html+css+js
  • Sends AJAX (XHR) requests to REST API
  • Browser uses that data to construct the DOM
  • Handles changes via events ( DOM loaded, mouse click)

Poor Firefox :(

CDP

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( fullUrl, {waitUntil: 'networkidle0'});

  await browser.close();
})();
  • Flake
  • Goto
  • WaitForPage
  • DOM Events

Pros

Cons (so far)

The wave?

  • Chrome taking over​
  • Evergreen Browsers
  • Web has matured

=

  • Cross-browser less important
  • Fewer browsers
  • Fewer browser bugs
  • Chrome drives change
  • Always stable interface

Lets cross these browsers

Full circle

Ansible

What browser version are you on right now?

What is Cross Browser Testing?

Browsers are evergreen

Modern Web Replaced

  • Thick Clients. Vcenter
  • Plugins are gone.
  • Rails, PHP
  • isflashdeadyet.com

"Modern" Javascript Frameworks

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)
  • Note:  upgrade path for enterprise customers is uncertain

i.e.

IE

Ride The Wave
 

(This message brought to you by Google)

What are you testing?

  • Browser versions (APIs)
  • Rendering engines
    • Webkit
    • Blink
    • Gecko
  • Does your app work with new browser APIs
  • Does your app work with each OS

Selenium = Cross Browser

Browser APIs rapidly changing

https://chromestatus.com/features/5707065825689600

The Matrix

Where does QA spend time?

  • Frameworks prioritize Time to Value to get new users up and running quickly
  • QA Spends our time analyzing nightly failures
  • Logs, Reproducing Locally, Video (if fortunate)
  • Authoring New Tests
  • Maintaining framework changes alongside Chrome

Cross-browser

Speed through parallelization

  • Selenium Grid is used for parallelization of tests
  • Hub and spoke architecture from the 90s
  • Grid leads to Hub...

Hub leads to

Cross x Browser

"We need Selenium because we need cross browser testing"

~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid