Flexible Data Setup And Caching For E2E Tests

Gleb Bahmutov, @bahmutov

Cypress.io

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end and we are close

https://www.ipcc.ch/assessment-report/ar6/

@bahmutov

survival is possible* but we need to act now

  • change your life
  • change your bank
  • join an organization
  • vote and protest

@bahmutov

rebellion.global www.wedonthavetime.org 

www.sierraclub.org 350.org www.greenpeace.org 

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

Chat application

https://github.com/bahmutov/chat.io

@bahmutov

Test:

Send and receive messages

Chat room

User

Create manually

Pass to the test runner

{
  "env": {
    "username": "gleb1",
    "password": "gleb1"
  }
}
# Cypress config and the spec files
const username = Cypress.env('username')
const password = Cypress.env('password')
cy.visit('/')
cy.get('.login-form').within(() => {
  cy.get('[placeholder=username]').type(username)
  cy.get('[placeholder=password]').type(password)
  cy.contains('button', 'login').click()
})
cy.location('pathname').should('equal', '/rooms')

cypress.json config file

cypress spec file

$ npx cypress open
# or pass the username and password via CLI arguments
$ npx cypress open --env username=gleb1,password=gleb1

Recommended: https://glebbahmutov.com/blog/keep-passwords-secret-in-e2e-tests/

🚨 Slow

🚨 Not repeatable

🚨 Hard to run tests in parallel

Create manually

Pass to the test runner

🚨 Slow

🚨 Not repeatable

🚨 Hard to run tests in parallel

Create manually

Pass to the test runner

How did I create this user? Ughh, does it need permission X set? Let me ask around...

🚨 Slow

🚨 Not repeatable

🚨 Hard to run tests in parallel

Create manually

Pass to the test runner

At MercariUS we only use env variables to pass basic auth that allows one to visit the deployed dev environment URL

const username = Cypress.env('username')
const password = Cypress.env('password')
cy.visit('/', {
  auth: {username, password}
})

Alternative: Create user in the test

it('registers user in the test', () => {
  const username = 'Test'
  const password = 'MySecreT'

  registerUser(username, password)
  loginUser(username, password)

  // if the user has been created and could log in
  // we should be redirected to the home page with the rooms
  cy.location('pathname').should('equal', '/rooms')
})

@bahmutov

  • ✅ First test run
  • 🚨 Second test run
it('creates a random user', () => {
  const username = 'Gleb-' + Cypress._.random(1e3)
  const password = '¡SoSecret!'
  registerUser(username, password)
  loginUser(username, password)
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
})

Generate using random data

The test should create all the data it needs

Tip: do not clean up the created data from the test itself.

If test data grows to be a problem, use a dedicated cron job to delete it

The test should create all the data it needs

  • ✅ Tests are independent of each other
  • ✅ Repeatable
  • 🚨 Might be slow

Dealing with caching edge-cases

5

6

Real-world testing success at MercariUS and Extend

The End

4

Dependencies between data sessions; sharing across specs

1

Introduction and the data creation problem

2

Creating and caching the user, the session, the room using custom code

3

Creating data using cypress-data-session plugin

@bahmutov

The test should create all the data it needs

once

let username
let password

beforeEach(() => {
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    registerUser(username, password)
  }
})

it('creates a random user once', () => {
  loginUser(username, password)
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
})

The test should create all the data it needs

once

let username
let password

beforeEach(() => {
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    registerUser(username, password)
  }
})

it('creates a random user once', () => {
  loginUser(username, password)
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
})

Not going to work

variables are reset during each test run

Use more permanent memory to store the created data

Cypress.env https://on.cypress.io/env

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

Use more permanent memory to store the created data

Cypress.env https://on.cypress.io/env

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

Look up the created user from DevTools

  • 🐢 create + login 4.67s
  • 🐎 login 2.3s

Reusing cached data makes the tests faster

How to log in instantly

When the server logs the user, it sends back the session cookie named "connect.sid"

Cypress can read and set cookies:

  • cy.getCookie
  • cy.setCookie
  • cy.clearCookie

@bahmutov

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

beforeEach(() => {
  if (Cypress.env('cookie')) {
    cy.setCookie('connect.sid', Cypress.env('cookie'))
    cy.visit('/')
  } else {
    loginUser(username, password)
    cy.getCookie('connect.sid').then((cookie) => {
      Cypress.env('cookie', cookie.value)
    })
  }
})

it('logs in using a cookie', () => {
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
})
  1. Create and cache the user as before
  2. Get and cache the cookie
  • 🐢 create + login 4.67s
  • 🐎 login 2.3s
  • 🚀 cookie 0.3s

Create a room if necessary

beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      .type(roomName)
      .next()
      .click()
    cy.contains('.room-item', roomName).should('be.visible')
    Cypress.env('room', roomName)
  }
})

@bahmutov

Pass the room name via alias

beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      .type(roomName)
      .next()
      .click()
    cy.contains('.room-item', roomName).should('be.visible')
    Cypress.env('room', roomName)
    cy.wrap(roomName).as('roomName')
  } else {
    // need to set the alias before each test
    cy.wrap(Cypress.env('room')).as('roomName')
  }
})

Use the alias in the test

beforeEach(() => {
  cy.wrap(roomName).as('roomName')
})

it('logs in using a cookie', function () {
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
  cy.contains('.room-item', this.roomName).click()
  cy.location('pathname').should('include', '/chat/')
  cy.contains('.chat-room', this.roomName).should('be.visible')
})

Use the alias in the test

beforeEach(() => {
  cy.wrap(roomName).as('roomName')
})

it('logs in using a cookie', function () {
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', username)
  cy.contains('.room-item', this.roomName).click()
  cy.location('pathname').should('include', '/chat/')
  cy.contains('.chat-room', this.roomName).should('be.visible')
})

Common patterns

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

⠔ checking memory

beforeEach(() => {
  if (Cypress.env('cookie')) {
    cy.setCookie('connect.sid', Cypress.env('cookie'))
    cy.visit('')
  } else {
    loginUser(username, password)
    cy.getCookie('connect.sid').then((cookie) => {
      Cypress.env('cookie', cookie.value)
    })
  }
})
beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      ...
    Cypress.env('room', roomName)
    cy.wrap(roomName).as('roomName')
  } else {
    // need to set the alias before each test
    cy.wrap(Cypress.env('room')).as('roomName')
  }
})

@bahmutov

Common patterns

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

⠜ "setup" vs "recreate"

beforeEach(() => {
  if (!Cypress.env('cookie')) {
    loginUser(username, password)
    cy.getCookie('connect.sid').then((cookie) => {
      Cypress.env('cookie', cookie.value)
    })
  } else {
    cy.setCookie('connect.sid', Cypress.env('cookie'))
    cy.visit('/')
  }
})
beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      ...
    Cypress.env('room', roomName)
    cy.wrap(roomName).as('roomName')
  } else {
    // need to set the alias before each test
    cy.wrap(Cypress.env('room')).as('roomName')
  }
})

Common patterns

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + Cypress._.random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})

⠜ saving values

beforeEach(() => {
  if (!Cypress.env('cookie')) {
    loginUser(username, password)
    cy.getCookie('connect.sid').then((cookie) => {
      Cypress.env('cookie', cookie.value)
    })
  } else {
    cy.setCookie('connect.sid', Cypress.env('cookie'))
    cy.visit('/')
  }
})
beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      ...
    Cypress.env('room', roomName)
    cy.wrap(roomName).as('roomName')
  } else {
    // need to set the alias before each test
    cy.wrap(Cypress.env('room')).as('roomName')
  }
})

cypress-data-session

https://github.com/bahmutov/cypress-data-session

let username
let password

beforeEach(() => {
  username = Cypress.env('username')
  password = Cypress.env('password')
  if (!username) {
    username = 'Gleb-' + random(1e3)
    password = '¡SoSecret!'
    Cypress.env('username', username)
    Cypress.env('password', password)
    registerUser(username, password)
  }
})
beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
  })
})

Cypress command for flexible test data setup

cypress-data-session@v2

@bahmutov

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
  })
})
beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
  })
})

it('has a user', function () {
  // each value is stored under the session name
  expect(this.user.password).to.equal('¡SoSecret!')   
})

The value yielded by the last setup command is cached in memory

cypress-data-session

https://github.com/bahmutov/cypress-data-session

beforeEach(() => {
  if (Cypress.env('cookie')) {
    cy.setCookie('connect.sid', Cypress.env('cookie'))
    cy.visit('')
  } else {
    loginUser(username, password)
    cy.getCookie('connect.sid').then((cookie) => {
      Cypress.env('cookie', cookie.value)
    })
  }
})
beforeEach(function () {
  cy.dataSession({
    name: 'login',
    setup() {
      loginUser(this.user.username, this.user.password)
      cy.getCookie('connect.sid').its('value')
    },
    recreate(value) {
      cy.setCookie('connect.sid', value)
      cy.visit('/')
    },
  })
})

Store the cookie using data session

First test run

creates the user

creates the session

Second test run

takes the saved user and session

cypress-data-session

https://github.com/bahmutov/cypress-data-session

beforeEach(() => {
  if (!Cypress.env('room')) {
    const roomName = 'Room-' + Cypress._.random(1e3)
    cy.get('input[aria-label="New room name"]')
      ...
    Cypress.env('room', roomName)
    cy.wrap(roomName).as('roomName')
  } else {
    // need to set the alias before each test
    cy.wrap(Cypress.env('room')).as('roomName')
  }
})
beforeEach(() => {
  cy.dataSession({
    name: 'roomName',
    setup() {
      const roomName = 'Room-' + random(1e3)
      cy.get('input[aria-label="New room name"]')
        .type(roomName)
        .next()
        .click()
      cy.wrap(roomName)
    },
  })
})

Store the created room using data session

it('uses data session', function () {
  cy.location('pathname').should('equal', '/rooms')
  cy.contains('.user-info', this.user.username)
  cy.contains('.room-item', this.roomName).click()
  cy.location('pathname').should('include', '/chat/')
  cy.contains('.chat-room', this.roomName).should('be.visible')
})

value from the last command in the "setup" method

See the data session value from DevTools

Hard browser reload ⌘-R clears Cypress.env

@bahmutov

  beforeEach(() => {
    cy.dataSession({
      name: 'user',
      setup() {
        const username = 'Gleb-' + random(1e3)
        const password = '¡SoSecret!'
        registerUser(username, password)
        cy.wrap({ username, password })
      },
+     shareAcrossSpecs: true
    })
  })
  cy.dataSession({
    name: 'login',
    setup() {
      loginUser(this.user.username, this.user.password)
      cy.getCookie('connect.sid').its('value')
    },
    recreate(value) {
      cy.setCookie('connect.sid', value)
      cy.visit('/')
    },
+   shareAcrossSpecs: true,
  })
  cy.dataSession({
    name: 'roomName',
    setup() {
      const roomName = 'Room-' + random(1e3)
      cy.get('input[aria-label="New room name"]')
        .type(roomName)
        .next()
        .click()
      cy.contains('.room-item', roomName).should('be.visible')
      cy.wrap(roomName)
    },
+   shareAcrossSpecs: true,
  })

shareAcrossSpecs: true 😄

data sessions

Cypress process

Browser process

Data can be cached in the browser or in the Cypress plugins process (shareAcrossSpecs: true)

Dealing with caching edge-cases

5

6

Real-world testing success at MercariUS and Extend

The End

4

Dependencies between data sessions; sharing across specs

1

Introduction and the data creation problem

2

Creating and caching the user, the session, the room using custom code

3

Creating data using cypress-data-session plugin

@bahmutov

Test:

Send and receive messages

Chat room

User

What if the user is no longer there?

@bahmutov

Delete the user session

Hmm, the "login" data session still uses the old cookie for the first user

  cy.dataSession({
    name: 'login',
    setup() {
      loginUser(this.user.username, this.user.password)
      cy.getCookie('connect.sid').its('value')
    },
    recreate(value) {
      cy.setCookie('connect.sid', value)
      cy.visit('/')
    },
    shareAcrossSpecs: true,
+   dependsOn: ['user']
  })

Use directed acyclical graph to recompute data sessions

Cached data vs real data

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
  })
})
  1. The user is created
  2. Another test deletes all users
  3. The test tries to use the user that no longer exists in the database 
  • ✅ memory
  • ❌ database

Need to verify the data first

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
    validate(user) {
      // check if the user is still valid
      // yield true | false
    }
  })
})

Need to verify the data first

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
    validate(user) {
      return cy.task('findUser', user.username)
    }
  })
})
// cypress/plugins/index.js
const database = require('../../app/database')

async function findUser(username) {
  console.log('find user', username)
  if (typeof username !== 'string') {
    throw new Error('username must be a string')
  }
  return database.models.user.findOne({ username })
}

on('task', {
  findUser
})

browser test file

Cypress plugins file

Need to verify the data first

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      const username = 'Gleb-' + random(1e3)
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
    validate(user) {
      return cy.task('findUser', user.username)
    }
  })
})

browser test file

Initialize from real data

beforeEach(() => {
  cy.dataSession({
    name: 'user',
    setup() {
      // no ramdom data, just the username
      const username = 'Gleb'
      const password = '¡SoSecret!'
      registerUser(username, password)
      cy.wrap({ username, password })
    },
    validate(user) {
      return cy.task('findUser', user.username)
    }
  })
})
  1. The user "Gleb" is created
  2. The next time Cypress is loaded, it tries to create the user "Gleb"
  3. The test fails - there is a user with the username "Gleb" in the database
  • ❌ memory
  • ✅ database

Initialize from real data

beforeEach(() => {
  // no ramdom data, just the username
  const username = 'Gleb'
  const password = '¡SoSecret!'
  cy.dataSession({
    name: 'user',
    init() {
      cy.task('findUser', username)
    },
    setup() {
      registerUser(username, password)
      cy.wrap({ username, password })
    },
    validate(user) {
      return cy.task('findUser', user.username)
    }
  })
})

cypress-data-session control flow

Take shortcuts for ⏱

  • Go through the page

  • Go through the API calls

  • Go through the server code / database

@bahmutov

tasks

Cypress process

Browser process

App api

Database

Take shortcuts

Testing the user registration

cy.visit('/')
cy.get('.register-form')
  .within(() => {
    cy.get('[placeholder=username]')
      .type(username)
    cy.get('[placeholder=password]')
      .type(password)
    cy.contains('button', 'register')
      .click()
  })

User data session

cy.dataSession({
  name: 'user',
  setup () {
    cy.visit('/')
    cy.get('.register-form')
      .within(() => {
        cy.get('[placeholder=username]')
          .type(username)
        cy.get('[placeholder=password]')
          .type(password)
        cy.contains('button', 'register')
          .click()
      })    
    cy.wrap({ username, password })
  }
})

@bahmutov

Take shortcuts

Testing the user registration

cy.visit('/')
cy.get('.register-form')
  .within(() => {
    cy.get('[placeholder=username]')
      .type(username)
    cy.get('[placeholder=password]')
      .type(password)
    cy.contains('button', 'register')
      .click()
  })

User data session

cy.dataSession({
  name: 'user',
  setup () {
    cy.task('makeUser', { username, password })
    cy.wrap({ username, password })
  }
})

Take shortcuts

Testing the user login

cy.visit('/')
cy.get('.login-form')
  .within(() => {
    cy.get('[placeholder=username]')
      .type(username)
    cy.get('[placeholder=password]')
      .type(password)
    cy.contains('button', 'login')
      .click()
  })
cy.location('pathname')
  .should('equal', '/rooms')
cy.contains('.user-info', username)

Login data session

cy.dataSession({
  name: 'login',
  setup() {
    cy.request({
      method: 'POST',
      url: '/login',
      form: true,
      body: {
        username: this.user.username,
        password: this.user.password,
      },
    })
    cy.getCookie('connect.sid').its('value')
  },
  recreate(value) {
    cy.setCookie('connect.sid', value)
  },
  shareAcrossSpecs: true,
  dependsOn: ['user'],
})

Take shortcuts

Testing the user login

Login data session

{Summary}

Creating complex data to use during tests is its own problem. 

@bahmutov

  • Speed
  • Use
  • Consistency

{Summary}

My solution is the cypress-data-session plugin

  • Speed
  • Use
  • Consistency

{Summary}

My solution is the cypress-data-session plugin

  • Speed

Only create the test data once, use API / database

  • Use
  • Consistency

{Summary}

My solution is the cypress-data-session plugin

Create aliases automatically

  • Speed
  • Use
  • Consistency

{Summary}

My solution is the cypress-data-session plugin

validate(), init(), recreate() + invalidation API

  • Speed
  • Use
  • Consistency

cypress-data-session is used at MercariUS

abstractions that use cy.dataSession

@bahmutov

cypress-data-session is used at MercariUS

during the test

cypress-data-session is used at Extend           https://www.extend.com/

https://dev.to/muratkeremozcan/crud-api-testing-a-deployed-service-with-cypress-using-cy-api-spok-cypress-data-session-cypress-each-4mlg

Thank You 👏

Gleb Bahmutov, @bahmutov

🖥 https://slides.com/bahmutov/data-for-e2e-tests

🎁 https://github.com/bahmutov/cypress-data-session

Examples, blog posts, videos