Life after Selenium

by John Hill

Ansible QE

About Me

In this talk

  • Selenium and Browsers
  • What are the problems with Selenium
  • "Ride the wave"
  • What are the new solutions
  • Walk through a test in 3 frameworks
  • What happens next
  • Q&A

In this talk

  • Convince you to delete and restart
  • Get commission
  • Mobile
  • All alternatives
  • python, java

Selenium

  • Open Source UI Testing tool to automate browser interactions for web applications.
  • Created in 2004. Maintained by SeleniumHQ.
  • Selenium = The infra + tooling to run webdriver
  • Webdriver = a part of the w3c browser spec.
    • Cross browser by definition!

Selenium Diagram

What problems did Selenium solve?

  • Testability for web browsers.
  • Open source! No more QTP!
  • Standardized interactions for all browsers via w3c webdriver spec.
  • Provide language bindings for those interactions
  • Provide infrastructure and tooling for the above

What is webdriver... driving?

Web Browser

  • Serves up web applications.
  • Sends and receives AJAX (XHR) requests to a backend (REST API)
  • That data constructs the DOM or the structure of the code on screen.
  • Browser handles changes via as w3c events ( DOM loaded, mouse click)

Demo Cypress Real World App with Selenium

Test Review

 

  1. Launch Chrome
  2. Navigate to RWA
  3. Check the title of the page
  4. Log in by filling in username / password and 'Login' button click
  5. Navigate to /transaction/new via button click
  6. Search for "Devon Becker" and verify result
    (http://localhost:3001/users/search?q=Devon+Becker)

Browser Review

  1. DOMComplete event
  2. Fires off XHR Requests to API
  3. API returns Response
  4. Events! DOM Loaded, Mouse Click Events
  5. Browser Renders or Repaints based on XHR return or events

Problems

Flake

  • "UI Testing is Flaky"
  • Difficult to understand WHY test failed
    • Video / Screenshots would have helped us!
  • Reliability baseline of 3% from Saucelabs

Observer

Flaky Demo

WTF (Why the flake)

  • NOT Event-driven
  • Polling for events vs Listening Events
  • "Loose coupling"
  • Lack of observability

"loose coupling"

Lots to Manage

  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version
  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version

Docker-selenium solves part of this... at a cost

HW Resources

  • Expensive to run
  • Jmeter example 1000:1 Ratio
  • Video Recording via VNC extremely expensive
    (+1 Core, +1 GB of Memory, Storage)

Speed / Performance

  • HTTP Traffic is heavily impacted by network latency
  • RWA IS LOCAL!!!
  • Red Hat Test
  • Comcast Chaos Engineering
  • ping ondemand.saucelabs.com

Problem Summary

  • Lots of dependencies
  • Speed / Performance
  • Expensive
  • Flaky due to "loose coupling"
  • ... Cross Browser?

Cross x Browser

Selenium = Cross Browser

Cross-browser

What are you testing?

  • Rendering engines
    • Webkit
    • Blink
    • Gecko
  • Does your app work with new browser APIs
  • Does your app work with each OS

Stale (as of 10/13/20)

Ride The Wave
 

(This message brought to you by Google)

Chrome

i.e.

IE

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)
  • Note:  upgrade path for enterprise customers is uncertain

Browsers are evergreen

What browser version are you on right now?

What is Cross Browser Testing?

The wave?

  • Chrome taking over​
  • Evergreen Browsers
  • Web has matured

=

  • Cross-browser less important
  • Fewer browsers
  • Fewer browser bugs
  • Chrome drives change
  • Always stable interface

"Solutions"

Puppeteer

  • "How Chrome tests Chrome"
  • Chrome DevTools Protocol (CDP) API
    • API matches webdriver
  • Evergreen NodeJS Lib+ Chromium bin.
    • Lightweight!
    • "npm install puppeteer"
  • Firefox support* in Nightlies
  • Not a Framework. Just a Selenium replacement.

Puppeteer (continued)

  • Event-driven
  • Websocket interface to chrome browser
    • Minimal Impact from Network Latency
  • CDP exposes Native Browser Interaction
    • Intercept and interact with network calls

Webdriver vs Puppeteer

CDP

Webdriver CDP
w3c standard Yes No
Interaction Polling Real-time
Dependency Heck Heaven
Cross-Browser Yes Sorta*
Inherently Flaky Yes No
"Weight" Heavy Light

Quick Summary

Webdriver.io

  • V1 on 12/2013
  • Framework
  • Webdriver-based. Not  selenium.
  • Auto-wait, auto-retry
  • 03/26/20 Puppeteer is now default!

Demo

  • "Cross-Browser testing done right"
  • v1 on Jan 31 2020
  • CDP API
  • Controversial
    • MS Hired Puppeteer team
    • Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
  • Can run WebKit (the upstream safari browser engine) on Windows and linux!
  • Not a total Framework. More of a fork of puppeteer. Easy to migrate to/from puppeteer.

 

Demo

  • V1 10/2017
  • Proprietary Proxy Server. Not Puppeteer
  • Freemium OSS*
  • More than a framework
    • All in One solution
    • Documentation!
    • GUI Playground
    • *Dashboard service
  • Cross-platform**

Demo

Cons (so far)

Pros

Solutions Summary

  • Real Browser interaction
  • Realtime
  • Fewer dependencies and maintenance
  • Still Cross Browser!

F U T U R E

CDP

Poor Firefox :(

Ansible

Full circle

Selenium 4.0

Summary

  • Selenium
  • Problems
  • "Rode the wave"
  • Solutions
  • Walked through 3 frameworks
  • What happens next

Cutting room floor

CDP

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( fullUrl, {waitUntil: 'networkidle0'});

  await browser.close();
})();
  • Flake
  • Goto
  • WaitForPage
  • DOM Events

"Modern" Javascript Frameworks

Modern Web Replaced

  • Thick Clients. Vcenter
  • Plugins are gone.
  • Rails, PHP
  • isflashdeadyet.com

Where does QA spend time?

  • Frameworks prioritize Time to Value to get new users up and running quickly
  • QA Spends our time analyzing nightly failures
  • Logs, Reproducing Locally, Video (if fortunate)
  • Authoring New Tests
  • Maintaining framework changes alongside Chrome

Speed through parallelization

  • Selenium Grid is used for parallelization of tests
  • Hub and spoke architecture from the 90s
  • Grid leads to Hub...

Hub leads to