Life after Selenium

by John Hill

Ansible QE

About Me

In this talk

  • Selenium and Browsers
  • What are the problems
  • "Ride the wave"
  • What are the new solutions
  • Walk through a test in 3 frameworks
  • What happens next
  • Q&A

In this talk

  • Convince you to delete and restart
  • Get commission
  • Mobile
  • All alternatives
  • python, java

Selenium

  • Open Source UI Testing tool to automate browser interactions for web applications.
  • Created in 2004. Maintained by SeleniumHQ.
  • Selenium (2020) = now the infra + tooling to run webdriver
  • Webdriver (2020) = now a part of the w3c browser spec.
  • Cross browser by definition!

Selenium Diagram

What is webdriver... driving?

What is a web browser?

  • Serves up web applications
  • Rendering Engine for Javascript (i.e. Chrome = Blink)
  • Browser receives Webapp code via html+css+js
  • Sends AJAX (XHR) requests to REST API
  • Browser uses that data to construct the DOM
  • Handles changes via events ( DOM loaded, mouse click)

Demo Cypress Real World App

Browser Review

  1. Browser downloads CSS+HTML+JS
  2. DOM Loaded Event
  3. Fires off XHR Requests to API
  4. Renders and Paints what it can
  5. Repaints based on XHR return or events

Cross x Browser

"We need Selenium because we need cross browser testing"

~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid

Cross-browser

Browser APIs rapidly changing

https://chromestatus.com/features/5707065825689600

The Matrix

Selenium = Cross Browser

What are you testing?

  • Browser versions (APIs)
  • Rendering engines
    • Webkit
    • Blink
    • Gecko
  • Does your app work with new browser APIs
  • Does your app work with each OS

Mo' Problems

Lots to Manage

  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version
  • Framework/Test Runner
  • Selenese Language Binding
  • Webdriver/binary Type and Version
  • Browser Type and Version
  • Browser Capabilities "API"
  • Selenium JAR
  • JRE
  • OS Type
  • OS Version

Docker-selenium solves part of this

Speed / Performance

  • HTTP Traffic is heavily impacted by network latency
  • Comcast Chaos Engineering
  • ping ondemand.saucelabs.com

Observer

HW Resources

  • Expensive to run
  • Jmeter example 1000:1 Ratio
  • Video Recording via VNC extremely expensive
    (+1 Core, +1 GB of Memory, Storage)

Cross browser

Flake

  • "UI Testing is Flaky"
  • Difficult to understand WHY test failed
  • Video / Screenshots help!
  • Reliability baseline of 3% from Saucelabs

WTF (Why the flake)

  • NOT Event-driven
  • Polling for events vs Listening Events
  • "Loose coupling"
  • Lack of observability

"loose coupling"

Flaky Demo

Problem Summary

  • Lots of dependencies without ownership
  • Speed / Performance
  • Expensive
  • Flaky due to "loose coupling"
  • Cross Browser?

Stale (as of 08/13/20)

Ride The Wave
 

(This message brought to you by Google)

Chrome

i.e.

IE

Edge Chromium

  • Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
  • Chromium is just Chrome without licensed codecs.
  • (You can't netflix and chill)
  • Note:  upgrade path for enterprise customers is uncertain

"Modern" Javascript Frameworks

Modern Web Replaced

  • Thick Clients. Vcenter
  • Plugins are gone.
  • Rails, PHP
  • isflashdeadyet.com

Browsers are evergreen

What browser version are you on right now?

What is Cross Browser Testing?

Lets cross these browsers

The wave?

  • Chrome taking over​
  • Evergreen Browsers
  • Web has matured

=

  • Cross-browser less important
  • Fewer browsers
  • Fewer browser bugs
  • Chrome drives change
  • Always stable interface

"Solutions"

Puppeteer

  • "How Chrome tests Chrome"
  • Chrome DevTools Protocol (CDP) API
    • API matches webdriver
  • Evergreen NodeJS Lib+ Chromium bin.
    • Cheap and 
  • Firefox support* in Nightlies
  • Not a Framework. Just a Selenium replacement.

Puppeteer (continued)

  • Event-driven
  • Websocket interface to chrome browser
    • Minimal Impact from Network Latency
  • CDP exposes Native Browser Interaction
    • Intercept and interact with network calls

Webdriver vs Puppeteer

CDP

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( fullUrl, {waitUntil: 'networkidle0'});

  await browser.close();
})();
  • Flake
  • Goto
  • WaitForPage
  • DOM Events

CDP

Webdriver.io

  • V1 on 12/2013
  • Framework
  • Webdriver-based. Not  selenium.
  • Auto-wait, auto-retry
  • 03/26/20 Puppeteer is now default!

Demo

  • "Cross-Browser testing done right"
  • v1 on Jan 31 2020
  • CDP API
  • Controversial
    • MS Hired Puppeteer team
    • Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
  • Can run WebKit (the upstream safari browser engine) on Windows and linux!
  • Not a total Framework. More of a fork of puppeteer. Easy to migrate to/from puppeteer.

 

Demo

  • V1 10/2017
  • Proprietary Proxy Server. Not Puppeteer
  • Freemium OSS*
  • More than a framework
    • All in One solution
    • Documentation!
    • GUI Playground
    • *Dashboard service
  • Cross-platform**
    • Firefox
    • Edge

Demo

Cons (so far)

Pros

Solutions Summary

  • Real Browser interaction
  • Realtime
  • Less dependencies and maintenance
  • Still Cross Browser!

F U T U R E

CDP

Poor Firefox :(

Ansible

Full circle

Selenium 4.0

Summary

  • Selenium
  • Problems
  • "Rode the wave"
  • Solutions
  • Walked through 3 frameworks
  • What happens next

Cutting room floor

Where does QA spend time?

  • Frameworks prioritize Time to Value to get new users up and running quickly
  • QA Spends our time analyzing nightly failures
  • Logs, Reproducing Locally, Video (if fortunate)
  • Authoring New Tests
  • Maintaining framework changes alongside Chrome

Speed through parallelization

  • Selenium Grid is used for parallelization of tests
  • Hub and spoke architecture from the 90s
  • Grid leads to Hub...

Hub leads to