Life after Selenium
by John Hill
Ansible QE
About Me
Durham
In this talk
- Selenium and Browsers
- What are the problems
- What are the new solutions
- Walk through a test in 3 frameworks
- What happens next
- Q&A
In this talk
Convince you to delete and restart-
Get commission MobileAll alternativespython, java
Selenium
- Open Source UI Testing tool to automate browser interactions for web applications.
- Created in 2004. Maintained by SeleniumHQ.
- Selenium (2020) = the infra + tooling to run webdriver
- Webdriver (2020) = part of the w3c browser spec
- Cross browser by definition!
Selenium Diagram
What is webdriver... driving?
Demo Cypress Real World App
Browser Review
- Browser downloads CSS+HTML+JS
- Fires off XHR Requests to API
- Browser waits!
- Rebuilds DOM. Repaints based on XHR return
Request: /users/search?q=devon+becker Receives JSON Payload: {"results": [{"firstName":"Devon","lastName":"Becker"
Problematic Selenium Demo
Mo' Problems
Lots to Manage
- Framework/Test Runner
- Selenese Language Binding
- Webdriver/binary Type and Version
- Browser Type and Version
- Browser Capabilities "API"
- Selenium JAR
- JRE
- OS Type
- OS Version
- Framework/Test Runner
- Selenese Language Binding
- Webdriver/binary Type and Version
- Browser Type and Version
- Browser Capabilities "API"
- Selenium JAR
- JRE
- OS Type
- OS Version
While preparing for today
Docker-selenium solves part of this
HW Resources
- Expensive to run
- Jmeter example 1000:1 Ratio
- Video Recording via VNC extremely expensive
(+1 Core, +1 GB of Memory, Storage)
Selenium as a Service $$$
Speed / Performance
- HTTP Traffic is heavily impacted by network latency
- Comcast Chaos Engineering
- ping ondemand.saucelabs.com = 75 ms
Same Test with 100ms of latency between Jenkins and Chrome
Flake
- "UI Testing is Flaky"
- Difficult to understand WHY test failed
- Video / Screenshots help!
- Reliability baseline of 3% from major SaaS provider
WTF (Why the flake)
- NOT Event-driven
- Polling for events vs Listening Events
- "Loose coupling"
- Lack of observability
"loose coupling"
Flaky Demo Fix
Problem Summary
- Lots of dependencies without ownership
- Speed / Performance
- Expensive
- Flaky due to "loose coupling"
Stale (as of 11/23/20)
"Solutions"
What's new since 2004?
i.e.
IE
Edge Chromium
- Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
- Chromium is just Chrome without licensed codecs.
- (You can't netflix and chill)
Puppeteer
- "How Chrome tests Chrome"
- Chrome DevTools Protocol (CDP) API
- API matches webdriver
- Evergreen NodeJS Lib+ Chromium bin.
- Cheap and lightweight
- Firefox support* in Nightlies
- Not a Framework. Just a Selenium replacement.
Puppeteer (continued)
- Event-driven
- Websocket interface to chrome browser
- Minimal Impact from Network Latency
- CDP exposes Native Browser Interaction
- Intercept and interact with network calls
Webdriver vs Puppeteer
CDP
Quick Summary
| Webdriver | CDP | |
|---|---|---|
| w3c standard | Yes | No |
| Interaction | Polling | Real-time |
| Dependency | Heck | Heaven |
| Cross-Browser | Yes | Sorta* |
| Inherently Flaky | Yes | No |
| "Weight" | Heavy | Light |
Webdriver.io
- V1 on 12/2013
- Framework
- Webdriver-based. Not selenium.
- Auto-wait, auto-retry
- 03/26/20 Puppeteer is now default!
Demo
- "Cross-Browser testing done right"
- v1 on Jan 31 2020
- CDP API
- Controversial
- MS Hired Puppeteer team
- Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
- Can run WebKit (the upstream safari browser engine) on Windows and linux!
- Not a framework! Fork of puppeteer. Easy to migrate to/from puppeteer.
Demo
- V1 10/2017
- Proprietary Proxy Server. Not CDP.
- Freemium OSS*
- More than a framework
- All in One solution
- Documentation!
- GUI Playground
- *Dashboard service
- Cross-platform**
- Firefox
- Edge
Demo
Where are we now?
Solutions Summary
- Real Browser interaction
- Realtime
- Fewer dependencies and maintenance
- Still Cross Browser!
F U T U R E
CDP
Selenium 4.0
- Your tests from 10 years ago probably still work
- *dramatic pause *
- github.com/SeleniumHQ/selenium/issues/8168
Summary
- Selenium
- Problems
- Solutions
- Walked through
32 frameworks - What happens next
Cutting room floor
What is a web browser?
- Serves up web applications
- Rendering Engine for Javascript (i.e. Chrome = Blink)
- Browser receives Webapp code via html+css+js
- Sends AJAX (XHR) requests to REST API
- Browser uses that data to construct the DOM
- Handles changes via events ( DOM loaded, mouse click)
Poor Firefox :(
CDP
const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto( fullUrl, {waitUntil: 'networkidle0'}); await browser.close(); })();
Flake- Goto
- WaitForPage
- DOM Events
Pros
Cons (so far)
- Proprietary
- iFrames
- Cannot orchestrate multiple tabs
- "Cost" of cross platform abstraction
- Link to our tests
The wave?
- Chrome taking over
- Evergreen Browsers
- Web has matured
=
- Cross-browser less important
- Fewer browsers
- Fewer browser bugs
- Chrome drives change
- Always stable interface
Lets cross these browsers
Full circle
Ansible
- Visual Testing
- Ansible AWX Testing
What browser version are you on right now?
What is Cross Browser Testing?
Browsers are evergreen
Modern Web Replaced
- Thick Clients. Vcenter
- Plugins are gone.
- Rails, PHP
- isflashdeadyet.com
"Modern" Javascript Frameworks
Edge Chromium
- Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
- Chromium is just Chrome without licensed codecs.
- (You can't netflix and chill)
- Note: upgrade path for enterprise customers is uncertain
i.e.
IE
Ride The Wave
(This message brought to you by Google)
What are you testing?
- Browser versions (APIs)
- Rendering engines
- Webkit
- Blink
- Gecko
- Does your app work with new browser APIs
- Does your app work with each OS
Browser APIs rapidly changing
The Matrix
Where does QA spend time?
- Frameworks prioritize Time to Value to get new users up and running quickly
- QA Spends our time analyzing nightly failures
- Logs, Reproducing Locally, Video (if fortunate)
- Authoring New Tests
- Maintaining framework changes alongside Chrome
Cross-browser
Speed through parallelization
- Selenium Grid is used for parallelization of tests
- Hub and spoke architecture from the 90s
- Grid leads to Hub...
Hub leads to
Cross x Browser
"We need Selenium because we need cross browser testing"
~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid
Life after Selenium by John Hill Ansible QE
Life After Selenium (11/25)
By John Hill
Life After Selenium (11/25)
- 741
