Life after Selenium
by John Hill
Ansible QE
About Me
In this talk
- Selenium and Browsers
- What are the problems
- "Ride the wave"
- What are the new solutions
- Walk through a test in 3 frameworks
- What happens next
- Q&A
In this talk
Convince you to delete and restart-
Get commission MobileAll alternativespython, java
Selenium
- Open Source UI Testing tool to automate browser interactions for web applications.
- Created in 2004. Maintained by SeleniumHQ.
- Selenium (2020) = now the infra + tooling to run webdriver
- Webdriver (2020) = now a part of the w3c browser spec.
- Cross browser by definition!
Selenium Diagram
What is webdriver... driving?
What is a web browser?
- Serves up web applications
- Rendering Engine for Javascript (i.e. Chrome = Blink)
- Browser receives Webapp code via html+css+js
- Sends AJAX (XHR) requests to REST API
- Browser uses that data to construct the DOM
- Handles changes via events ( DOM loaded, mouse click)
Demo Cypress Real World App
Browser Review
- Browser downloads CSS+HTML+JS
- DOM Loaded Event
- Fires off XHR Requests to API
- Renders and Paints what it can
- Repaints based on XHR return or events
Cross x Browser
"We need Selenium because we need cross browser testing"
~A QA Manager who is not ultimately budgeting the cost of maintaining the Selenium Grid
Cross-browser
Browser APIs rapidly changing
The Matrix
What are you testing?
- Browser versions (APIs)
- Rendering engines
- Webkit
- Blink
- Gecko
- Does your app work with new browser APIs
- Does your app work with each OS
Mo' Problems
Lots to Manage
- Framework/Test Runner
- Selenese Language Binding
- Webdriver/binary Type and Version
- Browser Type and Version
- Browser Capabilities "API"
- Selenium JAR
- JRE
- OS Type
- OS Version
- Framework/Test Runner
- Selenese Language Binding
- Webdriver/binary Type and Version
- Browser Type and Version
- Browser Capabilities "API"
- Selenium JAR
- JRE
- OS Type
- OS Version
Docker-selenium solves part of this
Speed / Performance
- HTTP Traffic is heavily impacted by network latency
- Comcast Chaos Engineering
- ping ondemand.saucelabs.com
Observer
HW Resources
- Expensive to run
- Jmeter example 1000:1 Ratio
- Video Recording via VNC extremely expensive
(+1 Core, +1 GB of Memory, Storage)
Cross browser
Flake
- "UI Testing is Flaky"
- Difficult to understand WHY test failed
- Video / Screenshots help!
- Reliability baseline of 3% from
Saucelabs
WTF (Why the flake)
- NOT Event-driven
- Polling for events vs Listening Events
- "Loose coupling"
- Lack of observability
"loose coupling"
Flaky Demo
Problem Summary
- Lots of dependencies without ownership
- Speed / Performance
- Expensive
- Flaky due to "loose coupling"
- Cross Browser?
Stale (as of 08/13/20)
Ride The Wave
(This message brought to you by Google)
Chrome
i.e.
IE
Edge Chromium
- Late 2018 announced a switch from EdgeHTML engine to Chrome's Blink
- Chromium is just Chrome without licensed codecs.
- (You can't netflix and chill)
- Note: upgrade path for enterprise customers is uncertain
"Modern" Javascript Frameworks
Modern Web Replaced
- Thick Clients. Vcenter
- Plugins are gone.
- Rails, PHP
- isflashdeadyet.com
Browsers are evergreen
What browser version are you on right now?
What is Cross Browser Testing?
Lets cross these browsers
The wave?
- Chrome taking over
- Evergreen Browsers
- Web has matured
=
- Cross-browser less important
- Fewer browsers
- Fewer browser bugs
- Chrome drives change
- Always stable interface
"Solutions"
Puppeteer
- "How Chrome tests Chrome"
- Chrome DevTools Protocol (CDP) API
- API matches webdriver
- Evergreen NodeJS Lib+ Chromium bin.
- Cheap and
- Firefox support* in Nightlies
- Not a Framework. Just a Selenium replacement.
Puppeteer (continued)
- Event-driven
- Websocket interface to chrome browser
- Minimal Impact from Network Latency
- CDP exposes Native Browser Interaction
- Intercept and interact with network calls
Webdriver vs Puppeteer
CDP
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto( fullUrl, {waitUntil: 'networkidle0'});
await browser.close();
})();Flake- Goto
- WaitForPage
- DOM Events
CDP
Webdriver.io
- V1 on 12/2013
- Framework
- Webdriver-based. Not selenium.
- Auto-wait, auto-retry
- 03/26/20 Puppeteer is now default!
Demo
- "Cross-Browser testing done right"
- v1 on Jan 31 2020
- CDP API
- Controversial
- MS Hired Puppeteer team
- Fork of Fork of firefox-puppeteer but not the same one that firefox currently supports
- Can run WebKit (the upstream safari browser engine) on Windows and linux!
- Not a total Framework. More of a fork of puppeteer. Easy to migrate to/from puppeteer.
Demo
- V1 10/2017
- Proprietary Proxy Server. Not Puppeteer
- Freemium OSS*
- More than a framework
- All in One solution
- Documentation!
- GUI Playground
- *Dashboard service
- Cross-platform**
- Firefox
- Edge
Demo
Cons (so far)
- Proprietary
- iFrames
- Cannot orchestrate multiple tabs
- "Cost" of cross platform abstraction
- Link to our tests
Pros
Solutions Summary
- Real Browser interaction
- Realtime
- Less dependencies and maintenance
- Still Cross Browser!
F U T U R E
CDP
Poor Firefox :(
Ansible
- Visual Testing
- Ansible AWX Testing
Full circle
Selenium 4.0
- Your tests from 10 years ago probably still work
- "No reason that webdriver selenium cannot operate with this new API" - SeleniumHQ person
- https://github.com/SeleniumHQ/selenium/issues/8168
Summary
- Selenium
- Problems
- "Rode the wave"
- Solutions
- Walked through 3 frameworks
- What happens next
Cutting room floor
Where does QA spend time?
- Frameworks prioritize Time to Value to get new users up and running quickly
- QA Spends our time analyzing nightly failures
- Logs, Reproducing Locally, Video (if fortunate)
- Authoring New Tests
- Maintaining framework changes alongside Chrome
Speed through parallelization
- Selenium Grid is used for parallelization of tests
- Hub and spoke architecture from the 90s
- Grid leads to Hub...
Hub leads to
Life After Selenium 07/08/20
By John Hill
