Testing Shapes and Paradigms

Gleb Bahmutov

Sr Director of Engineering

Roman Sandler

Frontend Developer

https://slides.com/bahmutov/the-testing-shapes

https://slides.com/romansandler/bulletproof-cypress/

https://www.youtube.com/watch?v=F8LH9d9eN3M

www.mercari.com/careers/

https://live.remo.co/e/mercari-global-engineers-meetup-1

Agenda

  • Testing Pyramid(s)

  • Speed matters. But which one?

  • The Testing Matrix

  • Iterative testing method

  • Q & A

The original pyramid

Mike Cohn -  Succeeding with Agile (2009)

The Pyramid transforms

Different communities adopt the idea and make adjustments to it.

The core idea remains

  • Higher level tests === More integration, slow and expensive.
  • Lower level tests === More isolation, fast and cheap.

Challenging the pyramid

Write tests, not too many, mostly integration

Kent C. Dodds - assert.js 2018

E2E

Testing Pyramid

with holes?

crash

reporting

linters

static

types

unit

Ramda, _

Find the differences

Same Different
Higher-level tests are slower and more expensive Most of your tests should be intergration level tests
End to End tests should have a small part of the total Unit tests should have a smaller part of the total

Main message: Compromise on the effort to gain more confidence.

in different pyramids...

More integration requires more effort

Effort

Integration

Lower level tests require a small effort

Effort

Level of abstraction

Higher level tests require a large effort

Effort

Level of abstraction

Well...

Let's re-examine this "truth"

Testing Tools Effort Simulator

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + Enzyme

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + RTL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Visual Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Selenium

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Any testing level "recipe" has 2 problems:

  1. They lack context

  2. They become a dogma

The Testing Matrix

Optimize for maximum confidence at minimum effort

Suggestion: Spiral Model

Iterate

Iterate

Iterate

Planning

Coding / Testing

Deploying

Maintaining 

Feature Iterations: Planning

  • What are the available resources for testing?

  • What testing tools are they currently familiar with?

  • What is the cost of a bug in the feature?

  • How important is it to verify the implementation details? 

Feature Iterations: Coding / Testing

  • Does it spark joy?

  • What's painful?

  • How long does it take?

  • How much code is it?

  • Is the test readable?

Feature Iterations: Deploying

  • How scared are you to deploy? On Friday?

  • What do you test manually?

  • What did you miss?

  • Did you go over budget? Or over the deadline?

Feature Iterations: Maintenance

  • How much flake do you accept?

  • Can the implementation change?

  • Do you hate the testing code?

In Summary 🎓

  • There can't be a "one size fits all" approach for testing.
  • The effort you put into your test suite must be worth the confidence it affords you.
  • Iterate, iterate, iterate. Revisit the decisions you've made thus far and be critical of them.
  • Don't be dogmatic, be pragmatic.

Thank you!

Gleb Bahmutov

Sr Director of Engineering

Roman Sandler

Frontend Developer

https://slides.com/bahmutov/the-testing-shapes

@bahmutov

@RomanSndlr