Gleb Bahmutov

Sr Director of Engineering

Andrae Ambrose

Enterprise Software Developer

# 21 Questions

Album Get Rich or Die Tryin' in 2009

# Agenda

1. What is a test?
2. Where should a test live?
3. How long should a test be?
4. Who should be writing tests?
5. When should you run your tests?
6. What are the signs of a bad test?
7. Do you have confidence in your tests?
8. Should you retry your tests on failure?
9. How do you pick a test runner?
10. How do you push for more testing?

# 1. What is a test?

## An independent confirmation: perform X and get Y

``expect(sum(2, 3)).to.equal(5)``
``````cy.get('#submit').click()
cy.contains('Success!').should('be.visible')``````

# 1. What is a test?

``expect(sum(2, 3)).to.equal(sum(4, 1))``

🚨

# 1. What is a test?

``````cy.get('#submit').should(_ => {})
.then(\$button => {
if (\$button.length) {
cy.wrap(\$button).click()
cy.contains('Success!')
.should('be.visible')
}
})``````

Logic in the test itself 🚩

## An independent confirmation: perform X and get Y

``expect(sum(2, 3)).to.equal(5)``

## An independent confirmation: perform X and get Y

``expect(sum(2, 3)).to.equal(5)``

## An independent confirmation: perform X and get Y

``````// maybe?
expect(sum(200, 300)).to.equal(500)
// possibly?
expect(sum(-2, -3)).to.equal(-5)
// unknown
expect(sum('two', 'three')).to.equal('five')
expect(sum(2)) // ?``````

# 2. Where should the test live?

A: With the code

B: In a separate repo

C: No-code solution

# A: With the code

• ✅ Zero friction
• ✅ Changes code AND changed tests
• 🔻 Extra tooling, typings, installation

# B: In a separate repo

• ✅ Good way to start writing tests
• 🔻 Maintenance nightmare

# C: No-code solution

• 🔻 Have not seen a successful example yet

# 3. How long should a test be?

## Most unit tests are very short

3. ### Assert

``expect(result).to.equal(5)``
``const result = sum(2, 3)``
``import {sum} from './math'``

# The test is too long

3. ## Starts next test from a "checkpoint"

``````cy.contains('Next').click()

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
.its('app.state')
.should('deep.equal', startOfSecondPageState)``````

# End of the first test

``````cy.window()
.its('app')
.invoke('setState', startOfSecondPageState)

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')

# 5. When should you run your tests?

A Developer should run his/her tests on every commit

Test early, test often.

# 6. What are the signs of a bad test?

``````//original spec
cy.get(`[data-testid-SearchInput]`).should("be.visible");
cy.get(`[data-testid-SignupButton]`).should("be.visible");
cy.get(`[data-testid-CategoryNav]`).should("be.visible");
});

//Refactored
const homePageSelectors = [

homePageSelectors.forEach((testid) => {
cy.get(`[data-testid=\${testid}]`).should("be.visible");
});
});``````

# 6. What are the signs of a bad test?

• Using highly brittle selectors that are subject to change.
``````<button
id="main"
class="btn btn-large"
name="submission"
role="button"
data-cy="submit"
>
Submit
</button>``````

# 6. What are the signs of a bad test?

•  Inconsistent results

FLAKE

# 7. Do you have confidence in your tests?

• What do you still test manually?
• What bugs have you missed?

YES AND NO!

# 9. How do you pick a test runner?

Tool X vs Tool Y

Running the most primitive "visit X" test case

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

CI Setup

The test runner

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

# The Testing Matrix

## Helps refine the testing strategy through iteration

source: TestJSSummit presentation by Roman Sandler and Gleb Bahmutov https://slides.com/bahmutov/pyramid-testjsummit

# 10. How do you push for more testing?

Speak Management's native language - Metrics and Money 📈 💰

Will Klein

"The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses."

— Michael Bolton, author, “Rapid Software Testing”

Gleb Bahmutov

Sr Director of Engineering

@bahmutov

Andrae Ambrose

Enterprise Software Developer

@andraeambrose