Life after Selenium (abbreviated)

by John Hill

Ansible QE

About Me

In this talk

  • Selenium and Browsers
  • What are the problems
  • What are the new solutions
  • Walk through a test in 32 frameworks
  • What happens next
  • Q&A

github.com/unlikelyzero/life-after-selenium

In this talk

  • Convince you to delete and restart
  • Get commission
  • Mobile
  • All alternatives
  • python, java

In the beginning...

QTP(now UTP)

  • Created By Mercury Interactive in 1998
  • First Browser UI Automation tools
  • Closed source. Proprietary.
  • $3995 per copy (in 1998)
  • Limited Browser support
  • Windows Only

Selenium

  • "Selenium is the cure for Mercury Poisoning"
  • Created in 2004. Now maintained by SeleniumHQ.
  • Open Source, Open Standard
  • Cross Browser by design!
  • Selenium (2021) = the infra + tooling to run webdriver
  • Webdriver (2021) = part of the w3c browser spec
  • Now Cross browser by definition!

While preparing for today...
"Open Standard"

What is webdriver... driving?

Demo Cypress Real World App

Browser Review

  1. Renders User Login Page
  2. Fires off AuthN XHR Requests to API Server
  3. Route to / (Home)
  4. Renders HomePage with 'New' button
  5. Routes Browser to /transaction/new
  6. Renders Searchbar.
  7. Fires off XHR API Requests:

     
  8. Displays results payload to user
     
Request: /users/search?q=devon+becker

Receives JSON Payload: {"results": [{"firstName":"Devon","lastName":"Becker"

Problematic Selenium Demo

WTF (Why the flake)

  • NOT Event-driven
  • Polling for events vs Listening Events
  • "Loose coupling"
  • Lack of observability

Same Test with 100ms of latency between Jenkins and Browser

Problem Summary

  1. Open. Lots of dependencies without ownership
  2. Abstraction.
    • Speed / Performance Issues
    • Expensive to run
    • Flaky due to "loose coupling"

"Solutions"

What's new since 2004?

Puppeteer

  • "How Chrome tests Chrome"
  • Chrome DevTools Protocol (CDP) API
    • API matches webdriver
  • Evergreen NodeJS Lib+ Chromium bin.
    • Cheap and lightweight
  • Firefox support* in Nightlies
  • Not a Framework. Just a Selenium replacement.

Puppeteer (continued)

  • Event-driven
  • Websocket interface to chrome browser
    • Minimal Impact from Network Latency
  • CDP exposes Native Browser Interaction
    • Intercept and interact with network calls

Webdriver vs Puppeteer

Quick Summary

Webdriver CDP
w3c standard Yes No
Interaction Polling Real-time
Dependency Heck Heaven
Cross-Browser Yes Sorta*
Inherently Flaky Yes No
"Weight" Heavy Light

Webdriver.io 

  • Best example of CDP's impact.
  • Puppeteer is now default!
    Not CDP, specifically.
  • Originally Webdriver-based.
    Not  selenium-based.
  • Open Source Framework
  • Auto-wait, auto-retry

Demo

  • "Cross-Browser testing done right"
  • Open Source*
  • Released by Microsoft on Jan 31 2020
  • CDP API shim
  • Controversial
    • MS Hired Puppeteer team
    • Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
  • Can run WebKit (the upstream safari browser engine) on Windows and linux!
  • Not a framework! Fork of puppeteer.
    Easy to migrate to/from puppeteer.

 

Demo

  • 1.0 on 10/2017
  • Proprietary Proxy Server. Not CDP.
  • Freemium OSS*
  • More than a framework
    • All in One solution
    • Documentation!
    • GUI Playground with IDE
    • *Dashboard service
  • Cross-platform**
    • Firefox
    • Edge

Demo

Where are we now with Cypress 6.0?

Solutions Summary

  • Realtime without Flake!
  • Fewer dependencies and maintenance
  • Cross Browser!
  • Open Source******

F U T U R E

CDP

Selenium 4.0

Summary

  • Selenium
  • Problems
  • Solutions
  • Walked through 32 frameworks
  • What happens next

Cutting room floor

Speed / Performance

  • HTTP Traffic is heavily impacted by network latency
  • Comcast Chaos Engineering
  • ping ondemand.saucelabs.com = 75 ms

CDP

HW Resources

  • Expensive to run
  • Jmeter example 1000:1 Ratio
  • Video Recording via VNC extremely expensive
    (+1 Core, +1 GB of Memory, Storage)

Selenium as a Service $$$

Observer

Flake

  • "UI Testing is Flaky"
  • Difficult to understand WHY test failed
  • Video / Screenshots help!
  • Reliability baseline of 3% from major SaaS provider

"loose coupling"

Flaky Demo Fix

Docker-selenium solves part of this

Mo' Problems

Lots to Manage

  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version
  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version

What is a web browser?

  • Serves up web applications
  • Rendering Engine for Javascript (i.e. Chrome = Blink)
  • Browser receives Webapp code via html+css+js
  • Sends AJAX (XHR) requests to REST API
  • Browser uses that data to construct the DOM
  • Handles changes via events ( DOM loaded, mouse click)

Poor Firefox :(

CDP

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( fullUrl, {waitUntil: 'networkidle0'});

  await browser.close();
})();
  • Flake
  • Goto
  • WaitForPage
  • DOM Events

Pros

Cons (so far)

The wave?

  • Chrome taking over​
  • Evergreen Browsers
  • Web has matured

=

  • Cross-browser less important
  • Fewer browsers
  • Fewer browser bugs
  • Chrome drives change
  • Always stable interface

Lets cross these browsers

Full circle

Ansible

What browser version are you on right now?

What is Cross Browser Testing?

Browsers are evergreen

Modern Web Replaced

  • Thick Clients. Vcenter
  • Plugins are gone.
  • Rails, PHP
  • isflashdeadyet.com

"Modern" Javascript Frameworks

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)
  • Note:  upgrade path for enterprise customers is uncertain

i.e.

IE

Ride The Wave
 

(This message brought to you by Google)

What are you testing?

  • Browser versions (APIs)
  • Rendering engines
    • Webkit
    • Blink
    • Gecko
  • Does your app work with new browser APIs
  • Does your app work with each OS

Selenium = Cross Browser

Browser APIs rapidly changing

https://chromestatus.com/features/5707065825689600

The Matrix

Where does QA spend time?

  • Frameworks prioritize Time to Value to get new users up and running quickly
  • QA Spends our time analyzing nightly failures
  • Logs, Reproducing Locally, Video (if fortunate)
  • Authoring New Tests
  • Maintaining framework changes alongside Chrome

Cross-browser

Speed through parallelization

  • Selenium Grid is used for parallelization of tests
  • Hub and spoke architecture from the 90s
  • Grid leads to Hub...

Hub leads to

Cross x Browser

"We need Selenium because we need cross browser testing"

~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid