Answers to Your

Questions About Testing

Gleb Bahmutov

Sr Director of Engineering

Andrae Ambrose

Enterprise Software Developer

50 Cent:

21 Questions

Album Get Rich or Die Tryin' in 2009

Agenda

  1. What is a test?
  2. Where should a test live?
  3. How long should a test be?
  4. Who should be writing tests?
  5. When should you run your tests?
  6. What are the signs of a bad test?
  7. Do you have confidence in your tests?
  8. Should you retry your tests on failure?
  9. How do you pick a test runner?
  10. How do you push for more testing?

1. What is a test?

An independent confirmation: perform X and get Y

expect(sum(2, 3)).to.equal(5)
cy.get('#submit').click()
cy.contains('Success!').should('be.visible')

1. What is a test?

expect(sum(2, 3)).to.equal(sum(4, 1))

An independent confirmation: perform X and get Y

🚨

1. What is a test?

cy.get('#submit').should(_ => {})
  .then($button => {
    if ($button.length) {
      cy.wrap($button).click()
      cy.contains('Success!')
        .should('be.visible')
    }
  })

Logic in the test itself 🚩

An independent confirmation: perform X and get Y

expect(sum(2, 3)).to.equal(5)

An independent confirmation: perform X and get Y

What about: perform Z?

expect(sum(2, 3)).to.equal(5)

An independent confirmation: perform X and get Y

// maybe?
expect(sum(200, 300)).to.equal(500)
// possibly?
expect(sum(-2, -3)).to.equal(-5)
// unknown
expect(sum('two', 'three')).to.equal('five')
expect(sum(2)) // ?

What about: perform Z?

Assumption: if a test consistently passes and matches what the user does -

when the user uses the app, the chances are high that it works as expected

2. Where should the test live?

A: With the code

B: In a separate repo

C: No-code solution

A: With the code

  • ✅ Zero friction
  • ✅ Changes code AND changed tests
  • 🔻 Extra tooling, typings, installation

B: In a separate repo

  • ✅ Good way to start writing tests
  • 🔻 Maintenance nightmare

C: No-code solution

  • 🔻 Have not seen a successful example yet

3. How long should a test be?

Most unit tests are very short

  1. Arrange

  2. Act

  3. Assert

expect(result).to.equal(5)
const result = sum(2, 3)
import {sum} from './math'

E2E Tests Should Be Meaningful

When a test runs for too long...

Multi-page form example

The test is too long

  1. Split into 3 tests

  2. End each test with a "checkpoint"

  3. Starts next test from a "checkpoint"

cy.contains('Next').click()

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
  .its('app.state')
  .should('deep.equal', startOfSecondPageState)

End of the first test

cy.window()
  .its('app')
  .invoke('setState', startOfSecondPageState)

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)

Start of the second test

Start a test right from the state at the end of the previous test

4. Who should be writing the tests?

Developer working on a feature X = write E2E tests for feature X

QA team automates all current manual E2E tests

5. When should you run your tests?

A Developer should run his/her tests on every commit

Test early, test often.

6. What are the signs of a bad test?

Repetition (DRY your code!)

//original spec
Cypress.Commands.add("loadHomePage", () => {
  cy.get(`[data-testid-SearchInput]`).should("be.visible");
  cy.get(`[data-testid-SignupButton]`).should("be.visible");
  cy.get(`[data-testid-LoginButton]`).should("be.visible");
  cy.get(`[data-testid-CategoryNav]`).should("be.visible");
});


//Refactored
Cypress.Commands.add("loadHomePage", () => {
  const homePageSelectors = [
    "SearchInput", "SignupButton", "LoginButton", "CategoryNav",];

  homePageSelectors.forEach((testid) => {
    cy.get(`[data-testid=${testid}]`).should("be.visible");
  });
});

6. What are the signs of a bad test?

  • Using highly brittle selectors that are subject to change.
<button
  id="main"
  class="btn btn-large"
  name="submission"
  role="button"
  data-cy="submit"
>
  Submit
</button>

6. What are the signs of a bad test?

  •  Inconsistent results

FLAKE

7. Do you have confidence in your tests?

  • What do you still test manually?
  • What bugs have you missed?

8. Should you retry your tests on failure?

YES AND NO!

9. How do you pick a test runner?

Tool X vs Tool Y

Running the most primitive "visit X" test case

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

 🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

⚠️

🚩

Testing Tools

Effort Simulator:

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

The test runner

No tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

No automated tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress with Gleb & Andrae

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Selenium

Installation

Writing tests

Running tests

Debugging tests

Maintenance

The Testing Matrix

Helps refine the testing strategy through iteration

source: TestJSSummit presentation by Roman Sandler and Gleb Bahmutov https://slides.com/bahmutov/pyramid-testjsummit

10. How do you push for more testing?

Speak Management's native language - Metrics and Money 📈 💰

Will Klein

"The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses."

 

— Michael Bolton, author, “Rapid Software Testing”

Gleb Bahmutov

Sr Director of Engineering

@bahmutov

Andrae Ambrose

Enterprise Software Developer

@andraeambrose

Thank You 👏

Answers to Your Burning Questions About Testing

By Gleb Bahmutov

Answers to Your Burning Questions About Testing

In the pursuit of building rock-solid end-to-end testing suites, there are many variables that need to be decided in order to maintain a consistent testing environment. If left undefined, these variables can lead to endless wasted cycles spent on the effects, rather than simply eliminating the root cause. Presented at EnterpriseNG Dec 2021, video at https://www.youtube.com/watch?v=_7SJ_sMAols

  • 2,452