Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
Testing Shapes and Paradigms
Gleb Bahmutov
Sr Director of Engineering
Roman Sandler
Frontend Developer
Agenda
-
Testing Pyramid(s)
-
Speed matters. But which one?
-
The Testing Matrix
-
Iterative testing method
-
Q & A
The original pyramid
Mike Cohn - Succeeding with Agile (2009)
The Pyramid transforms
Different communities adopt the idea and make adjustments to it.
The core idea remains
- Higher level tests === More integration, slow and expensive.
- Lower level tests === More isolation, fast and cheap.
Challenging the pyramid
Write tests, not too many, mostly integration
Kent C. Dodds - assert.js 2018
E2E
Testing Pyramid
with holes?
crash
reporting
linters
static
types
unit
Ramda, _
Find the differences
| Same | Different |
|---|---|
| Higher-level tests are slower and more expensive | Most of your tests should be intergration level tests |
| End to End tests should have a small part of the total | Unit tests should have a smaller part of the total |
Main message: Compromise on the effort to gain more confidence.
in different pyramids...
More integration requires more effort
Effort
Integration
Lower level tests require a small effort
Effort
Level of abstraction
Higher level tests require a large effort
Effort
Level of abstraction
Well...
Let's re-examine this "truth"
Testing Tools Effort Simulator
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest + Enzyme
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest + RTL
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest Snapshots
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Visual Snapshots
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Cypress
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Selenium
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Any testing level "recipe" has 2 problems:
-
They lack context
-
They become a dogma
The Testing Matrix
Optimize for maximum confidence at minimum effort
Suggestion: Spiral Model
Iterate
Iterate
Iterate
Planning
Coding / Testing
Deploying
Maintaining
Feature Iterations: Planning
-
What are the available resources for testing?
-
What testing tools are they currently familiar with?
-
What is the cost of a bug in the feature?
-
How important is it to verify the implementation details?
Feature Iterations: Coding / Testing
-
Does it spark joy?
-
What's painful?
-
How long does it take?
-
How much code is it?
-
Is the test readable?
Feature Iterations: Deploying
-
How scared are you to deploy? On Friday?
-
What do you test manually?
-
What did you miss?
-
Did you go over budget? Or over the deadline?
Feature Iterations: Maintenance
-
How much flake do you accept?
-
Can the implementation change?
-
Do you hate the testing code?
In Summary 🎓
- There can't be a "one size fits all" approach for testing.
- The effort you put into your test suite must be worth the confidence it affords you.
- Iterate, iterate, iterate. Revisit the decisions you've made thus far and be critical of them.
- Don't be dogmatic, be pragmatic.
Thank you!
Gleb Bahmutov
Sr Director of Engineering
Roman Sandler
Frontend Developer
The Testing Shapes: Pyramid, Trophy, and Crab
By Gleb Bahmutov
The Testing Shapes: Pyramid, Trophy, and Crab
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications. Presented at Mercari Dev meetup.
