Don’t Make These Testing Mistakes
Gleb Bahmutov
Distinguished Engineer
Cypress.io
@bahmutov
Climate crisis: we need to act now
- change your life
- join an organization
Agenda
-
Common mistakes ⛔️
-
wrong test level
-
not enough assertions
-
negative assertions
-
Documentation 📚
-
We got issues 😡
-
test length
-
synchronous access
-
writing Node code
Cypress: Common Mistakes
Cypress Tests
Run
In The
Browser!
// cypress/integration/spec.js
// NOT GOING TO WORK
const fs = require('fs')
fs.readFileSync(...)⛔️ Cannot simply access the file system
Access Node and OS
- cy.readFile
- cy.writeFile
- cy.task
- cy.exec
most powerful
// cypress/plugins/index.js
const Knex = require('knex')
const knexConfig = require('../../knexfile')
const { Model } = require('objection')
const knex = Knex(knexConfig.development)
Model.knex(knex)
const Person = require('../../models/Person')
/**
* @type {Cypress.PluginConfig}
*/
module.exports = (on, config) => {
on('task', {
resetPeopleTable() {
console.log('reset People table')
return Person.query().truncate()
},
})
}Plugin file runs in Node
See https://github.com/cypress-io/cypress-realworld-app for good cy.task example
API test
UI test
Pick the right test
Testing
Test type
User flow through the web app
Individual piece of code
Server backend
Visual page appearance
Visual test
https://on.cypress.io/visual-testing
Pick the right test
Testing
Test type
Individual framework component
Component test
https://on.cypress.io/component-testing
Responsive design
E2E tests with different viewports
https://on.cypress.io/viewport
Accessability
Node code
Cypress Node test runner
(stay tuned)
Tests that are too short
Tests that are too long
The Test Length
describe('my form', () => {
before(() => {
cy.visit('/users/new')
cy.get('#first').type('johnny')
})
it('has validation attr', () => {
cy.get('#first').should('have.attr', 'data-validation', 'required')
})
it('has active class', () => {
cy.get('#first').should('have.class', 'active')
})
it('has formatted first name', () => {
cy.get('#first').should('have.value', 'Johnny') // capitalized first letter
})
})Way too short!
The Test Length
describe('my form', () => {
before(() => {
cy.visit('/users/new')
})
it('validates and formats first name', () => {
cy.get('#first')
.type('johnny')
.should('have.attr', 'data-validation', 'required')
.and('have.class', 'active')
.and('have.value', 'Johnny')
})
})Recommended
The Test That's Too Long
💡 Split Tests > 30 seconds
- More productive local development 😀
- Faster problem detection 🤷♂️
- More opportunity for CI parallelization 🏎
- Fewer CI crashes 😉
Cypress commands are queued
it('test', () => {
let username = undefined
cy.visit('https://app.com')
cy.get('.user-name')
.then(($el) => {
// this line evaluates when .then executes
username = $el.text()
})
if (username) {
cy.contains(username).click()
} else {
// this will always run
// because username will always
// evaluate to undefined
cy.contains('My Profile').click()
}
})queued to be executed
Cypress commands are queued
it('test', () => {
let username = undefined
cy.visit('https://app.com')
cy.get('.user-name')
.then(($el) => {
// this line evaluates when .then executes
username = $el.text()
})
if (username) {
cy.contains(username).click()
} else {
// this will always run
// because username will always
// evaluate to undefined
cy.contains('My Profile').click()
}
})queued to be executed
Cypress commands are queued
it('test', () => {
let username = undefined
cy.visit('https://app.com')
cy.get('.user-name')
.then(($el) => {
// this line evaluates when .then executes
username = $el.text()
})
if (username) {
cy.contains(username).click()
} else {
// this will always run
// because username will always
// evaluate to undefined
cy.contains('My Profile').click()
}
})never runs
queued to be executed
Cypress commands are queued
it('test', () => {
cy.visit('https://app.com')
cy.get('.user-name')
.then(($el) => {
// this line evaluates when .then executes
const username = $el.text()
if (username) {
cy.contains(username).click()
} else {
cy.contains('My Profile').click()
}
})
})- the page is visited
- we find .user-name
- depending on its text we click
Not Enough Assertions
context('Navigation', () => {
it('can navigate around the website', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.get('main:contains("About")');
cy.get('[data-cy="header-link-users"]').click();
cy.get('main h1:contains("Users")');
});
});navigation test
Home Page ➡ About Page ➡ Users Page
Not Enough Assertions
💡 use cy.pause to step through the commands
Red flags 🚩🚩
The command found the wrong "About" text!
it('can navigate around the website (better)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.location('pathname').should('match', /\/about$/);
cy.contains('main h1', 'About').should('be.visible');
});- command
it('can navigate around the website (better)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.location('pathname').should('match', /\/about$/);
cy.contains('main h1', 'About').should('be.visible');
});- command
- assertion
it('can navigate around the website (better)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.location('pathname').should('match', /\/about$/);
cy.contains('main h1', 'About').should('be.visible');
});- command
- assertion
- command
💡 Alternate commands and assertions to make sure the test and the app behave as expected
💡 Be careful with negative assertions
// positive
cy.get('.todo-item')
.should('have.length', 2)
.and('have.class', 'completed')
// negative
cy.contains('first todo').should('not.have.class', 'completed')
cy.get('#loading').should('not.be.visible')Loading element goes away (wrong)
it('hides the loading element', () => {
cy.visit('/')
cy.get('.loading').should('not.be.visible')
})
Red 🚩: negative assertion passes
before the XHR
Loading element goes away (wrong)
it('uses negative assertion and passes for the wrong reason', () => {
cy.visit('/?delay=3000')
cy.get('.loading').should('not.be.visible')
})
Negative assertions passes even before the Ajax request starts
💡 Positive then negative assertion
it('use positive then negative assertion (flakey)', () => {
cy.visit('/?delay=3000')
// first, make sure the loading indicator shows up
cy.get('.loading').should('be.visible')
// then assert it goes away (negative assertion)
cy.get('.loading').should('not.be.visible')
})
💡 Slow down Ajax request
it('slows down the network response (works)', () => {
cy.intercept('/todos', {
body: [],
delayMs: 5000
})
cy.visit('/?delay=3000')
// first, make sure the loading indicator shows up
cy.get('.loading').should('be.visible')
// then assert it goes away (negative assertion)
cy.get('.loading').should('not.be.visible')
})
Documentation
If you are not reading it, you are making a huge mistake
Documentation makes or breaks projects
Every time I read our support forums and chats ... I feel pain
Every question here is a failure of the documentation
Every question here is probably a:
- a support issue 💵
- maybe a lost customer 💸
You Want Users to Find Answers to Their Questions by Themselves
The documentation demands for different user personas are contradictory
The Good Docs Are Hard
First time visitor
First time user
Repeat user
Paying customer
Promoter
💥
"show me Hello, World!"
"show me the changelog diff from version X to Y"
"show me a tutorial"
"show me how to do X"
"how do I solve my issue or bug?"
👀 Optimize documentation structure for beginners
- "Hello World" example front and center
- There are 10 beginners for every 1 advanced user. Every user starts as a beginner
👀 Optimize documentation structure for beginners
Then add all your documentation to the site reachable from the index
💡 Make a powerful docs search for every persona
Larger sites: Multiple roots
blog
examples
💡 Look at searches with no results
write docs for these searches!
💡 Search from CLI
Presentations about documentation search
Documentation Goals 🎯
Create 10x more Cypress examples and recipes
Keeping Examples Correct and Up-to-date
If I see a question on a forum, GitHub, Twitter, etc
How do I toggle a checkbox?
App HTML
Test code
Markdown file
Markdown file
Markdown spec also becomes a static docs page
Every example is scraped into the docs index
💡 Do NOT answer support questions
💡💡💡 Especially for private support
💡 Do NOT answer support questions
Instead, update the documentation, or create an example, or write a blog post.
Then answer with a link
💡 The Support Team
Should eliminate their own jobs by writing more and more documentation until the users find everything themselves
Agenda
-
Common mistakes ⛔️
-
wrong test level
-
not enough assertions
-
negative assertions
-
Documentation 📚
-
We got issues 😡
-
test length
-
synchronous access
-
writing Node code
We Got Issues 🔥
😡 Why don't you solve my problem!
(how to open a good GitHub issue)
😡 Why don't you solve my problem!
(how to open a good GitHub issue)
⛲️ Every day I'm hustlin'
about 30 hours of emails
⁉️ If you have a problem:
- Search our documentation
- Search open GitHub issues
- Look at the topics under label "topic: ..."
- is there a topic matching your problem?
- are there open or closed issues matching your issue?
🎁 If opening a new GH issue
- If we can reproduce it 👏📈
- FILL
- THE
You do not have to have fixtures / support / plugins
💡 Tip: remove what is not needed
To write good tests ✅
read the docs 📚
Thank you 👏
Gleb Bahmutov
Distinguished Engineer
Cypress.io
@bahmutov