E____

R_____

Pick Any Two

M______

E____

R_____

Pick Any Two

"You need the monitoring system to be substantially more reliable than the failure prone system or you don't gain reliability, just complexity."

- some reddit user

Rockets!

Tests must be more reliable than the code being tested

(that's surprisingly hard to do)

What tests promise to do

• Prevent costly bugs
• Accelerate software development

What tests Usually do

• Distract/impede developers
• Cause lots of false failures
• Get ignored
• Make for great hackathon presentations

How do you test that a car starts?

What does it take to run your test?

A fully working car!

• Gas
• Starter
• Spark plugs
• Belts
• Electronics/sensors
• Charged battery
• The right key!

How do you test a website works?

What does it take to run your test?

A fully working website!

• Browser/Device
• UI (HTML/CSS/JS)
• Server
• API
• Database
• Proper Data (search results for that word)
• Visual/Styles
• Time of day
• Third-party services/utilities
• Geographic region

M______

E____

R_____

Pick Any Two

M______

E____

R_____

Pick Any Two

M______

E____

R_____

Pick Any Two

What's the problem?

"Come, make UI testing work for us!"

• A sprint or two building out basic testing suite
• If lucky, integrated into CICD process
• Pieced together & built fast

AKA, the start of the End-to-End Death Spiral

A dark secret

"Why are our tests always failing?"

• Useful changes (improvements/tweaks)
• HTML structure/selector updates
• API changes
• Data changes
• Test flakiness issues (pauses and waits failing)
• Time changes
• Maybe bugs being caught?

Many layers = difficulty pinpointing change

Constant catch-up

"Getting behind the aircraft"

• Problems take a long time to figure out
• As you fix one problem, two new ones emerge
• As problems add up, you get further from the root cause, slowing down debugging even more
• Test usefulness plummets
• MAYDAY! MAYDAY!

A Testing failure

"Nice to have, but not for us"

Because testing was not properly invested in...

• All these failing tests create massive friction
• Friction creates massive frustration
• Frustration leads to ignoring results
• Ignoring results leads to out-of-date tests
• Out-of-date tests cause more failures
• Test suite crashes and burns

M______

E____

R_____

M______

E____

Reliability

High Confidence, High Reliability

• Ad hoc servers isolated from unrelated changes
• Test fixture setup/tear down
  • API integration
• Dedicated resources building and maintaining test suite
• Fully integrated into CICD process
• "Shift-left testing"

M______

Easy

Reliability

Reduce scope of testing

• Test fewer layers (mocking)
• Isolate what's being tested (e.g., Storybook)

Write UI Tests, not End-to-End tests

• Test the server on the server, not the browser

• Treat back-end as black box

• UI tests only check that user actions "work"

Only test important things

• You absolutely do not have to write tests for everything
• Focus on value gained from the test

Only test easy things

• Some things are *really* hard to test (e.g., animations)
• Just don't test them!
• Is it worth the time?

Mocks/Components

Easy

Reliability

An "Easy" Solution

GUI Based test tools

Mocks/Components

Easy

Reliability

Mocks/Components

Easy

Reliability

Mocks/Components

Easy

Reliability

Mocks/Components

Easy

Reliability

Mix and Match!

What about sstid?

• We have a lot of E2E tests
  • These are used to test the server

What about sstid?

• Migration towards UI/Component tests?
  • Don't test everything in every use case
  • Focus on "User Journeys"

What about sstid?

• Validating server functionality...
  • We still should do this somehow
• Testing integration between UI and server

What about sstid?

• What are our high value/critical items?
• How can get more value from our tests?