Testing Pyramid Makes Little Sense

Gleb Bahmutov

Sr Director of Engineering

Roman Sandler

Frontend Developer

And What We Can Use Instead

Agenda

  • Testing pyramid(s)

  • Speed matters. But which one?

  • The testing matrix

  • The testing spiral method

  • Q & A

The original pyramid

Mike Cohn -  Succeeding with Agile (2009)

The Pyramid transforms

Different communities adopt the idea and make adjustments to it.

The core idea remains

  • Higher level tests === More integration, slow and expensive.
  • Lower level tests === More isolation, fast and cheap.

Challenging the pyramid

Write tests, not too many, mostly integration

Kent C. Dodds - assert.js 2018

E2E

Testing Pyramid

with holes?

crash

reporting

linters

static

types

unit

Ramda, _

source: https://slides.com/bahmutov/assertjs

Find the differences

Same Different
Higher-level tests are slower and more expensive Most of your tests should be intergration level tests
End to End tests should have a small part of the total Unit tests should have a smaller part of the total

Main message: Compromise on the effort to gain more confidence.

in different pyramids...

Lower level tests require a small effort

Effort

Level of abstraction

More integration requires more effort

Effort

Integration

Higher level tests require a large effort

Effort

Level of abstraction

Well...

Let's re-examine this "truth"

Testing Tools Effort Simulator

Running tests

Tool X vs Tool Y

Running the most primitive "visit X" test case

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

 🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

Testing Tools

Effort Simulator:

Installation

Writing tests

Running tests

Debugging tests

Maintenance

What Is Missing?

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

The test runner

No tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

No automated tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + Enzyme

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + RTL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Visual Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Selenium

Installation

Writing tests

Running tests

Debugging tests

Maintenance

MABL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Any testing level "recipe" has 2 problems:

  1. They lack context

  2. They become a dogma

The Testing Matrix

Optimize for maximum confidence at minimum effort

Suggestion: Spiral Model

Iterate

Iterate

Iterate

Planning

Coding / Testing

Deploying

Maintaining 

Planning

Coding / Testing

Deploying

Feature Iterations: Planning

  • What are the available resources for testing?

  • What testing tools are they currently familiar with?

  • What is the cost of a bug in the feature?

  • How important is it to verify the implementation details? 

Questions to ask

Feature Iterations: Coding / Testing

  • Does it spark joy?

  • What's painful?

  • How long does it take?

  • How much code is it?

  • Is the test readable?

Questions to ask

Feature Iterations: Deploying

  • How scared are you to deploy? On Friday?

  • What do you test manually?

  • What did you miss?

  • Did you go over budget? Or over the deadline?

Questions to ask

Feature Iterations: Maintenance

  • How much flake do you accept?

  • Can the implementation change?

  • Do you hate the testing code?

Questions to ask

The Testing Matrix

Helps refine the testing strategy through iteration

sprint 1

sprint 2

sprint 3

sprint 4

sprint 5

In Summary 🎓

  • There can't be a "one size fits all" approach for testing.
  • The effort you put into your test suite must be worth the confidence it affords you.
  • Iterate, iterate, iterate. Revisit the decisions you've made thus far and be critical of them.
  • Don't be dogmatic, be pragmatic.

Thank you!

Gleb Bahmutov

Sr Director of Engineering

Roman Sandler

Frontend Developer

https://slides.com/bahmutov/pyramid-testjsummit

@bahmutov

@RomanSndlr