Who am I?

I don't work for:

• Zalando
• Aerokube
• Browserstack
• Saucelabs
• Testingbot


 

What is the "unit" of Selenium to scale?

Browser

Wrap the browser in a Containerized Selenium image

Selenium Docker Images

seleniumhq/Docker-selenium  
2973 Github Stars, 10M+ DockerHub Pulls
 

elgalu/docker-selenium
1100 Github Stars, 10M+ DockerHub Pulls
 

aerokube/selenoid
​913 Github Stars, 3M Docker Pulls

What's inside?

Small Detour

Dependency Heck

• House of Cards
• Hot issues
  • Se can't run on Java9
• Issues uncovered building demo
  • selenium/issues/5674
  • elgalu/docker-selenium/issues/201

"It works on my machine"

What does it mean to scale selenium testing?

Parallelism

Demo

• Serial
• Parallel
• Remote Parallel
• Remote Parallel++

Selenium Scaling Constraints

CPU & RAM

• ~ 1 CPU/Thread per session
• ~ 1000m in k8s/Openshift
• ~1 GB of RAM per session
• (More if recording video)

Network

• Selenium traffic is http traffic
• Latency between each component directly affects the speed of a test
• Each Test generates between 1-5 Requests Per Second (RPS)
• Up to 20 RPS

Selenium in CI

Anatomy of a Selenium Test in CI

Test Sharding

Scaling Path A

Cons

• Constrained by resource size of test executor
• Contesting other CI resources

Pros

• No network latency
• Local = CI

External Selenium Cluster

Scaling Path B

Cons

• Network latency

Pros

• Local = CI. Literally.
• Run locally "at scale"
• Separated concern
• Independent resources
• Many teams and frameworks can share resources

What does it take to create a cluster of selenium nodes?

Selenium Grid

• 2008
• 1 Hub for Many Nodes
• Single target for all tests

Who has maintained a Selenium Grid before?

Problems with Se Grid

• Single Point of Failure
• Not Highly Available
• No Load Balancing Strategy
• Hub must be configured, installed and hosted
• No "Autodiscovery" of Se Nodes
• Parasitic Network Traffic
• Dependency Heck++
• Lack of Observability
• Not Fault Tolerant

Zalenium

• zalando/zalenium
• Github Stars 1485, 5M+ Dockerhub Pulls
• Designed to work with elgalu/docker-selenium
• Dockerized Selenium Grid on Steroids

Zalenium "Creates" Docker-Selenium Nodes

Pros

• All-in-one solution
• "Elastic" Nodes
• Works as a generic Hub with any Se Node
• Proxies to SaaS
• Native k8s/Openshift support
• Pay-as-you-go
• Observability
• Scales to 150 Nodes

Cons

• Not Highly Available
• Not Fault Tolerant
• Java-based
• Scales to 150 Nodes

Zalenium "Proxies" to Se-as-a-Service

GoGridRouter

• aerokube/ggr
• Golang version of GridRouter
• 187 Github Stars, 105k Docker Pulls
• An active load balancer for Selenium traffic

GGR

Pros

• Intelligent Load balancing
• Works with any Selenium Hubs
• Works with SaaS
• Enables Highly Available clusters

Cons

• Not Fault Tolerant
• Not using standard selenium code
• Multiple GGR Nodes and a LB needed for HA
• Issues with 80% utilization

GoGridRouter

Moon

• aerokube/moon
• k8s native
• Selenoid+GGR

Pros

• Helm Chart config
• Perfect Load distribution
• Elastic!
• Pay-as-you-go
• 1000 Sessions with single cluster

Cons

• Closed Source
• Not FOSS
• Not Fault Tolerant

Saas

• Generic SaaS Pros apply
• Browserstack
• Saucelabs
• Testingbot

Cons

• Network Latency
• Issues at scale
• Costs

Costs

• Saucelabs
  • $75-$200 / session / month
  • ~$140,000 / month for 1000 Sessions
• Moon
  • $5 / session / month
  • k8s + $5,000 /month for 1000 Sessions
• Zalenium
  • $0.00017 to 0.00118 / minute / session
  • $36 + $34.67 / session / month

Selenium 4

When?

• Christmas 2018
• Chinese New Year 2019
• SeleniumHQ/selenium/projects/2
• (Opinion) Mired by backwards compatibility
• All new Grid Features may not land at launch

Grid 2.0

To complicate matters, the world has moved on since we wrote the original grid implementations. When we started writing grid, machines were underpowered, had limited memory, and SSDs didn’t really exist. Virtual machines were provided by VMWare and getting any real density of these required absolutely huge servers that few could afford. Consequently, we used multiple machines, self-hosted in racks rooms and local data centres. This lead to a really simple design for grid, which has allowed it to continue functioning relatively well to this day." - Simon Stewart

Grid 4.0

• Rewrite
• 7 Components
  • Edge Router + Distributor = GGR
  • Session Storage = Fault Tolerance via Redis
  • New Session Queuing System = AWS Lambda

Should we wait?