Don’t Make These Testing Mistakes

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov

Climate crisis: we need to act now

  • change your life
  • join an organization

rebellion.global          350.org

Agenda

  • Common mistakes ⛔️

  • wrong test level

  • not enough assertions

  • negative assertions

  • Documentation 📚

  • We got issues 😡

  • test length

  • synchronous access

  • writing Node code

Cypress: Common Mistakes

Cypress Tests

Run

In The

Browser!

// cypress/integration/spec.js
// NOT GOING TO WORK
const fs = require('fs')
fs.readFileSync(...)

⛔️ Cannot simply access the file system

Access Node and OS

  • cy.readFile
  • cy.writeFile
  • cy.task
  • cy.exec

most powerful

// cypress/plugins/index.js
const Knex = require('knex')
const knexConfig = require('../../knexfile')
const { Model } = require('objection')
const knex = Knex(knexConfig.development)
Model.knex(knex)
const Person = require('../../models/Person')

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  on('task', {
    resetPeopleTable() {
      console.log('reset People table')
      return Person.query().truncate()
    },
  })
}

Plugin file runs in Node

See https://github.com/cypress-io/cypress-realworld-app for good cy.task example 

API test

UI test

Pick the right test

Testing

Test type

User flow through the web app

Individual piece of code

Server backend

Visual page appearance

Pick the right test

Testing

Test type

Individual framework component

Responsive design

E2E tests with different viewports
https://on.cypress.io/viewport

Accessability

Node code

Cypress Node test runner
(stay tuned)

Tests that are too short

Tests that are too long

The Test Length

describe('my form', () => {
  before(() => {
    cy.visit('/users/new')
    cy.get('#first').type('johnny')
  })

  it('has validation attr', () => {
    cy.get('#first').should('have.attr', 'data-validation', 'required')
  })

  it('has active class', () => {
    cy.get('#first').should('have.class', 'active')
  })

  it('has formatted first name', () => {
    cy.get('#first').should('have.value', 'Johnny') // capitalized first letter
  })
})

Way too short!

The Test Length

describe('my form', () => {
  before(() => {
    cy.visit('/users/new')
  })

  it('validates and formats first name', () => {
    cy.get('#first')
      .type('johnny')
      .should('have.attr', 'data-validation', 'required')
      .and('have.class', 'active')
      .and('have.value', 'Johnny')
  })
})

Recommended

The Test That's Too Long

💡 Split Tests > 30 seconds

  • More productive local development 😀
  • Faster problem detection 🤷‍♂️
  • More opportunity for CI parallelization 🏎
  • Fewer CI crashes 😉

Cypress commands are queued

it('test', () => {
  let username = undefined

  cy.visit('https://app.com')
  cy.get('.user-name')
    .then(($el) => {
      // this line evaluates when .then executes
      username = $el.text()
    })
  
  if (username) {
    cy.contains(username).click()
  } else {
    // this will always run
    // because username will always
    // evaluate to undefined
    cy.contains('My Profile').click()
  }
})

queued to be executed

Cypress commands are queued

it('test', () => {
  let username = undefined

  cy.visit('https://app.com')
  cy.get('.user-name')
    .then(($el) => {
      // this line evaluates when .then executes
      username = $el.text()
    })
  
  if (username) {
    cy.contains(username).click()
  } else {
    // this will always run
    // because username will always
    // evaluate to undefined
    cy.contains('My Profile').click()
  }
})

queued to be executed

Cypress commands are queued

it('test', () => {
  let username = undefined

  cy.visit('https://app.com')
  cy.get('.user-name')
    .then(($el) => {
      // this line evaluates when .then executes
      username = $el.text()
    })
  
  if (username) {
    cy.contains(username).click()
  } else {
    // this will always run
    // because username will always
    // evaluate to undefined
    cy.contains('My Profile').click()
  }
})

never runs

queued to be executed

Cypress commands are queued

it('test', () => {
  cy.visit('https://app.com')
  cy.get('.user-name')
    .then(($el) => {
      // this line evaluates when .then executes
      const username = $el.text()
      if (username) {
        cy.contains(username).click()
      } else {
        cy.contains('My Profile').click()
      }
    })
})
  1. the page is visited
  2. we find .user-name
  3. depending on its text we click

Not Enough Assertions

context('Navigation', () => {
  it('can navigate around the website', () => {
    cy.visit('http://localhost:3000');

    cy.get('[data-cy="header-link-about"]').click();
    cy.get('main:contains("About")');

    cy.get('[data-cy="header-link-users"]').click();
    cy.get('main h1:contains("Users")');
  });
});

navigation test

Home Page ➡ About Page ➡ Users Page

Not Enough Assertions

💡 use cy.pause to step through the commands

https://on.cypress.io/pause

Red flags 🚩🚩

The command found the wrong "About" text!

it('can navigate around the website (better)', () => {
  cy.visit('http://localhost:3000');

  cy.get('[data-cy="header-link-about"]').click();
  cy.location('pathname').should('match', /\/about$/);
  cy.contains('main h1', 'About').should('be.visible');
});
  • command
it('can navigate around the website (better)', () => {
  cy.visit('http://localhost:3000');

  cy.get('[data-cy="header-link-about"]').click();
  cy.location('pathname').should('match', /\/about$/);
  cy.contains('main h1', 'About').should('be.visible');
});
  • command
  • assertion
it('can navigate around the website (better)', () => {
  cy.visit('http://localhost:3000');

  cy.get('[data-cy="header-link-about"]').click();
  cy.location('pathname').should('match', /\/about$/);
  cy.contains('main h1', 'About').should('be.visible');
});
  • command
  • assertion
  • command

💡 Alternate commands and assertions to make sure the test and the app behave as expected

💡 Be careful with negative assertions

// positive
cy.get('.todo-item')
  .should('have.length', 2)
  .and('have.class', 'completed')
// negative
cy.contains('first todo').should('not.have.class', 'completed')
cy.get('#loading').should('not.be.visible')

Loading element goes away (wrong)

it('hides the loading element', () => {
  cy.visit('/')
  cy.get('.loading').should('not.be.visible')
})

Red 🚩: negative assertion passes

before the XHR

Loading element goes away (wrong)

it('uses negative assertion and passes for the wrong reason', () => {
  cy.visit('/?delay=3000')
  cy.get('.loading').should('not.be.visible')
})

Negative assertions passes even before the Ajax request starts

💡 Positive then negative assertion

it('use positive then negative assertion (flakey)', () => {
  cy.visit('/?delay=3000')
  // first, make sure the loading indicator shows up
  cy.get('.loading').should('be.visible')
  // then assert it goes away (negative assertion)
  cy.get('.loading').should('not.be.visible')
})

💡 Slow down Ajax request

it('slows down the network response (works)', () => {
  cy.intercept('/todos', {
    body: [],
    delayMs: 5000
  })
  cy.visit('/?delay=3000')
  // first, make sure the loading indicator shows up
  cy.get('.loading').should('be.visible')
  // then assert it goes away (negative assertion)
  cy.get('.loading').should('not.be.visible')
})

Documentation

If you are not reading it, you are making a huge mistake

Documentation makes or breaks projects

Every time I read our support forums and chats ... I feel pain

Every question here is a failure of the documentation

Every question here is probably a:

  1. a support issue 💵
  2. maybe a lost customer 💸

You Want Users to Find Answers to Their Questions by Themselves

The documentation demands for different user personas are contradictory

The Good Docs Are Hard

First time visitor

First time user

Repeat user

Paying customer

Promoter

💥

"show me Hello, World!"

"show me the changelog diff from version X to Y"

"show me a tutorial"

"show me how to do X"

"how do I solve my issue or bug?"

👀 Optimize documentation structure for beginners

  • "Hello World" example front and center
  • There are 10 beginners for every 1 advanced user. Every user starts as a beginner

👀 Optimize documentation structure for beginners

Then add all your documentation to the site reachable from the index

💡 Make a powerful docs search for every persona

Larger sites: Multiple roots

blog

examples

💡 Look at searches with no results

write docs for these searches!

💡 Search from CLI

Presentations about documentation search

Documentation Goals 🎯

Create 10x more Cypress examples and recipes

Keeping Examples Correct and Up-to-date

If I see a question on a forum, GitHub, Twitter, etc

How do I toggle a checkbox?

App HTML

Test code

Markdown file

Markdown file

Markdown spec also becomes a static docs page

Every example is scraped into the docs index

💡 Do NOT answer support questions

💡💡💡 Especially for private support

💡 Do NOT answer support questions

Instead, update the documentation, or create an example, or write a blog post.

Then answer with a link

💡 The Support Team

Should eliminate their own jobs by writing more and more documentation until the users find everything themselves

Agenda

  • Common mistakes ⛔️

  • wrong test level

  • not enough assertions

  • negative assertions

  • Documentation 📚

  • We got issues 😡

  • test length

  • synchronous access

  • writing Node code

We Got Issues 🔥

😡 Why don't you solve my problem!

(how to open a good GitHub issue)

😡 Why don't you solve my problem!

(how to open a good GitHub issue)

⛲️ Every day I'm hustlin'

about 30 hours of emails

⁉️ If you have a problem:

  • Search our documentation
  • Search open GitHub issues
  • Look at the topics under label "topic: ..."
    • is there a topic matching your problem? 
    • are there open or closed issues matching your issue?

🎁 If opening a new GH issue

You do not have to have fixtures / support / plugins

💡 Tip: remove what is not needed

To write good tests ✅

read the docs 📚

Thank you 👏

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov

Don’t Make These Testing Mistakes

By Gleb Bahmutov

Don’t Make These Testing Mistakes

In this talk, I will discuss the common mistakes developers make when writing Cypress tests and how to avoid them. We will discuss tests that are too short, tests that hardcode data, tests that race against the application, and other mistakes. I believe this presentation will be useful to anyone writing E2E tests using JavaScript. Presented at TestJSSummit in Jan 2021

  • 3,195