open e2e testing initiative

Github Discussion Link

in this talk

• how to model the performance of a selenium test in ci/cd
• how to modify the test architecture of your ui testing

this is a selenium test

traditional test pipeline

1. ci trigger (i.e. nightly)
2. spin up ci worker (i.e. jenkins agent)
3. download test framework + code
4. spin up browser
5. spin up application under test
6. run tests

traditional "optimization"

• spin up components in parallel
  • spin up ci, application, browser at the same time
• pre-bake images
• optimize test code

Title Text

• Bullet One
• Bullet Two
• Bullet Three

comcast

find the network

impact of network in ui testing

• local vs comcast
• network bandwidth and latency
• "colocate work"
• how would we colocate a ui test?

totally colocated

vertical scaling

• everything run on same host?
• vertical scaling = more CPU and RAM
  • each executor needs CPU and RAM
  • each browser need 1 Core and 512 MB RAM
  • 364 Cores and 2 TB RAM?
• "problems" above >80% CPU and >90% RAM

horizontal scaling

• distribute the labor, comrades!
• add more
  • ci workers
  • runners
  • browsers (selenium grid)
  • frontend under test
  • backends?
• adding network again...

so what CAN we do?

REAL optimizations

1. model and trace your test pipelines
2. do as much as you can in the API
   • jmeter is 1000:1 CPU cost
3. colocate what you can i.e.
   • jenkins agents in same aws zone as prod
   • split the frontend and backend
   • trade resources with other teams!
4. nightlies are for sleeping
   • optimize for stability not runtime
   • run full matrix at night
   • optimize for fast feedback during day
   • i.e. run ONLY chrome against many frontends with a remote backend
5. run fewer tests
   • ​Launchable predictive test selection

summary

• model your ui tests!
• rethink ui testing performance!