Life after Selenium

by John Hill

Ansible QE

About Me

Durham

In this talk

  • Selenium and Browsers
  • What are the problems
  • What are the new solutions
  • Walk through a test in 3 frameworks
  • What happens next
  • Q&A

github.com/unlikelyzero/life-after-selenium

In this talk

  • Convince you to delete and restart
  • Get commission
  • Mobile
  • All alternatives
  • python, java

Selenium

  • Open Source UI Testing tool to automate browser interactions for web applications.
  • Created in 2004. Maintained by SeleniumHQ.
  • Selenium (2020) = the infra + tooling to run webdriver
  • Webdriver (2020) = part of the w3c browser spec
  • Cross browser by definition!

Selenium Diagram

What is webdriver... driving?

Demo Cypress Real World App

Browser Review

  1. Browser downloads CSS+HTML+JS
  2. Fires off XHR Requests to API
  3. Browser waits!
  4. Rebuilds DOM. Repaints based on XHR return
Request: /users/search?q=devon+becker

Receives JSON Payload: {"results": [{"firstName":"Devon","lastName":"Becker"

Problematic Selenium Demo

Mo' Problems

Lots to Manage

  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version
  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version

While preparing for today

Docker-selenium solves part of this

HW Resources

  • Expensive to run
  • Jmeter example 1000:1 Ratio
  • Video Recording via VNC extremely expensive
    (+1 Core, +1 GB of Memory, Storage)

Selenium as a Service $$$

Speed / Performance

  • HTTP Traffic is heavily impacted by network latency
  • Comcast Chaos Engineering
  • ping ondemand.saucelabs.com = 75 ms

Observer

Same Test with 100ms of latency between Jenkins and Chrome

Flake

  • "UI Testing is Flaky"
  • Difficult to understand WHY test failed
  • Video / Screenshots help!
  • Reliability baseline of 3% from major SaaS provider

WTF (Why the flake)

  • NOT Event-driven
  • Polling for events vs Listening Events
  • "Loose coupling"
  • Lack of observability

"loose coupling"

Flaky Demo Fix

Problem Summary

  1. Lots of dependencies without ownership
  2. Speed / Performance
  3. Expensive
  4. Flaky due to "loose coupling"

Stale (as of 11/23/20)

"Solutions"

What's new since 2004?

i.e.

IE

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)

Puppeteer

  • "How Chrome tests Chrome"
  • Chrome DevTools Protocol (CDP) API
    • API matches webdriver
  • Evergreen NodeJS Lib+ Chromium bin.
    • Cheap and lightweight
  • Firefox support* in Nightlies
  • Not a Framework. Just a Selenium replacement.

Puppeteer (continued)

  • Event-driven
  • Websocket interface to chrome browser
    • Minimal Impact from Network Latency
  • CDP exposes Native Browser Interaction
    • Intercept and interact with network calls

Webdriver vs Puppeteer

CDP

Quick Summary

Webdriver CDP
w3c standard Yes No
Interaction Polling Real-time
Dependency Heck Heaven
Cross-Browser Yes Sorta*
Inherently Flaky Yes No
"Weight" Heavy Light

Webdriver.io

  • V1 on 12/2013
  • Framework
  • Webdriver-based. Not  selenium.
  • Auto-wait, auto-retry
  • 03/26/20 Puppeteer is now default!

Demo

  • "Cross-Browser testing done right"
  • v1 on Jan 31 2020
  • CDP API
  • Controversial
    • MS Hired Puppeteer team
    • Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
  • Can run WebKit (the upstream safari browser engine) on Windows and linux!
  • Not a framework! Fork of puppeteer. Easy to migrate to/from puppeteer.

 

Demo

  • V1 10/2017
  • Proprietary Proxy Server. Not CDP.
  • Freemium OSS*
  • More than a framework
    • All in One solution
    • Documentation!
    • GUI Playground
    • *Dashboard service
  • Cross-platform**
    • Firefox
    • Edge

Demo

Where are we now?

Solutions Summary

  • Real Browser interaction
  • Realtime
  • Fewer dependencies and maintenance
  • Still Cross Browser!

F U T U R E

CDP

Selenium 4.0

Summary

  • Selenium
  • Problems
  • Solutions
  • Walked through 32 frameworks
  • What happens next

Cutting room floor

What is a web browser?

  • Serves up web applications
  • Rendering Engine for Javascript (i.e. Chrome = Blink)
  • Browser receives Webapp code via html+css+js
  • Sends AJAX (XHR) requests to REST API
  • Browser uses that data to construct the DOM
  • Handles changes via events ( DOM loaded, mouse click)

Poor Firefox :(

CDP

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( fullUrl, {waitUntil: 'networkidle0'});

  await browser.close();
})();
  • Flake
  • Goto
  • WaitForPage
  • DOM Events

Pros

Cons (so far)

The wave?

  • Chrome taking over​
  • Evergreen Browsers
  • Web has matured

=

  • Cross-browser less important
  • Fewer browsers
  • Fewer browser bugs
  • Chrome drives change
  • Always stable interface

Lets cross these browsers

Full circle

Ansible

What browser version are you on right now?

What is Cross Browser Testing?

Browsers are evergreen

Modern Web Replaced

  • Thick Clients. Vcenter
  • Plugins are gone.
  • Rails, PHP
  • isflashdeadyet.com

"Modern" Javascript Frameworks

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)
  • Note:  upgrade path for enterprise customers is uncertain

i.e.

IE

Ride The Wave
 

(This message brought to you by Google)

What are you testing?

  • Browser versions (APIs)
  • Rendering engines
    • Webkit
    • Blink
    • Gecko
  • Does your app work with new browser APIs
  • Does your app work with each OS

Selenium = Cross Browser

Browser APIs rapidly changing

https://chromestatus.com/features/5707065825689600

The Matrix

Where does QA spend time?

  • Frameworks prioritize Time to Value to get new users up and running quickly
  • QA Spends our time analyzing nightly failures
  • Logs, Reproducing Locally, Video (if fortunate)
  • Authoring New Tests
  • Maintaining framework changes alongside Chrome

Cross-browser

Speed through parallelization

  • Selenium Grid is used for parallelization of tests
  • Hub and spoke architecture from the 90s
  • Grid leads to Hub...

Hub leads to

Cross x Browser

"We need Selenium because we need cross browser testing"

~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid

Life After Selenium (11/25)

By John Hill

Life After Selenium (11/25)

  • 636

More from John Hill