Answers to Your
Questions About Testing
Gleb Bahmutov
Sr Director of Engineering
Andrae Ambrose
Enterprise Software Developer
50 Cent:
21 Questions
Album Get Rich or Die Tryin' in 2009
Fun fact: the song has 18 + 4 = 22 questions
https://www.billboard.com/articles/columns/hip-hop/8098488/50-cent-21-questions-lyrics
Agenda
- What is a test?
- Where should a test live?
- How long should a test be?
- Who should be writing tests?
- When should you run your tests?
- What are the signs of a bad test?
- Do you have confidence in your tests?
- Should you retry your tests on failure?
- How do you pick a test runner?
- How do you push for more testing?
1. What is a test?
An independent confirmation: perform X and get Y
expect(sum(2, 3)).to.equal(5)cy.get('#submit').click()
cy.contains('Success!').should('be.visible')1. What is a test?
expect(sum(2, 3)).to.equal(sum(4, 1))An independent confirmation: perform X and get Y
🚨
1. What is a test?
cy.get('#submit').should(_ => {})
.then($button => {
if ($button.length) {
cy.wrap($button).click()
cy.contains('Success!')
.should('be.visible')
}
})Logic in the test itself 🚩
An independent confirmation: perform X and get Y
expect(sum(2, 3)).to.equal(5)An independent confirmation: perform X and get Y
What about: perform Z?
expect(sum(2, 3)).to.equal(5)An independent confirmation: perform X and get Y
// maybe?
expect(sum(200, 300)).to.equal(500)
// possibly?
expect(sum(-2, -3)).to.equal(-5)
// unknown
expect(sum('two', 'three')).to.equal('five')
expect(sum(2)) // ?What about: perform Z?
Assumption: if a test consistently passes and matches what the user does -
when the user uses the app, the chances are high that it works as expected
2. Where should the test live?
A: With the code
B: In a separate repo
C: No-code solution
A: With the code
- ✅ Zero friction
- ✅ Changes code AND changed tests
- 🔻 Extra tooling, typings, installation
B: In a separate repo
- ✅ Good way to start writing tests
- 🔻 Maintenance nightmare
C: No-code solution
- 🔻 Have not seen a successful example yet
3. How long should a test be?
Most unit tests are very short
-
Arrange
-
Act
-
Assert
expect(result).to.equal(5)const result = sum(2, 3)import {sum} from './math'E2E Tests Should Be Meaningful
When a test runs for too long...
Multi-page form example
The test is too long
-
Split into 3 tests
-
End each test with a "checkpoint"
-
Starts next test from a "checkpoint"
cy.contains('Next').click()
cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
.its('app.state')
.should('deep.equal', startOfSecondPageState)End of the first test
cy.window()
.its('app')
.invoke('setState', startOfSecondPageState)
cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)Start of the second test
Start a test right from the state at the end of the previous test
4. Who should be writing the tests?
Developer working on a feature X = write E2E tests for feature X
QA team automates all current manual E2E tests
5. When should you run your tests?
A Developer should run his/her tests on every commit
Test early, test often.
6. What are the signs of a bad test?
Repetition (DRY your code!)
//original spec
Cypress.Commands.add("loadHomePage", () => {
cy.get(`[data-testid-SearchInput]`).should("be.visible");
cy.get(`[data-testid-SignupButton]`).should("be.visible");
cy.get(`[data-testid-LoginButton]`).should("be.visible");
cy.get(`[data-testid-CategoryNav]`).should("be.visible");
});
//Refactored
Cypress.Commands.add("loadHomePage", () => {
const homePageSelectors = [
"SearchInput", "SignupButton", "LoginButton", "CategoryNav",];
homePageSelectors.forEach((testid) => {
cy.get(`[data-testid=${testid}]`).should("be.visible");
});
});6. What are the signs of a bad test?
- Using highly brittle selectors that are subject to change.
<button
id="main"
class="btn btn-large"
name="submission"
role="button"
data-cy="submit"
>
Submit
</button>
6. What are the signs of a bad test?
- Inconsistent results
FLAKE
7. Do you have confidence in your tests?
- What do you still test manually?
- What bugs have you missed?
8. Should you retry your tests on failure?
YES AND NO!
9. How do you pick a test runner?
Tool X vs Tool Y
Running the most primitive "visit X" test case
⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩
⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩 ⚠️ 🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
⚠️
🚩
Testing Tools
Effort Simulator:
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
CI Setup
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
CI Setup
The test runner
No tests
Installation
Writing tests
Running tests
Debugging tests
Maintenance
No automated tests
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Cypress
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Cypress with Gleb & Andrae
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Selenium
Installation
Writing tests
Running tests
Debugging tests
Maintenance
The Testing Matrix
Helps refine the testing strategy through iteration
source: TestJSSummit presentation by Roman Sandler and Gleb Bahmutov https://slides.com/bahmutov/pyramid-testjsummit
10. How do you push for more testing?
Speak Management's native language - Metrics and Money 📈 💰
Will Klein
"The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses."
— Michael Bolton, author, “Rapid Software Testing”