Testing Tools and Their Friends

Gleb Bahmutov

Cypress.io

Boston, USA

these slides

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

24 people. Atlanta, Philly, Boston, NYC, Nashville,

Myanmar, Colombia, Ukraine

Fast, easy and reliable testing for anything that runs in a browser

You need crash reporting service

NOW

You need crash reporting service

everywhere

Even with crash reporting, you still lose airplanes

image source: http://ktla.com/2016/04/02/small-plane-crashes-into-suv-on-15-freeway-in-san-diego-county/

undefined is not a function

We going to need some tests

@bahmutov

https://glebbahmutov.com/blog/tags/testing/

E2E

integration

unit

Smallest pieces

@bahmutov

E2E

integration

unit

@bahmutov

E2E

integration

unit

Web application

@bahmutov

  • Open real browser
  • Load actual app
  • Interact with app like a real user
  • See if it works

E2E

integration

unit

@bahmutov

Really important to users

Really important to developers

npm i -D cypress
docker run cypress/included

or download Cypress application

1

example project: https://github.com/cypress-io/jsnation-example

or use Docker image

2

Write E2E test

it('adds and completes todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)

  cy.get('.todo').first().find('.toggle')
    .check()
  cy.get('.todo').first()
    .should('have.class', 'completed')
})

3

Run it on CI

version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          # we need to start the web application
          start: npm start

https://github.com/cypress-io/circleci-orb

https://circleci.com/gh/cypress-io/workflows/jsnation-example/tree/master

Tip: use start-server-and-test utility to start server

{
  "scripts": {
    "test": "cypress:run",
    "start": "parcel serve public/index.html",
    "e2e": "start-test 1234"
  }
}

https://github.com/bahmutov/start-server-and-test

Do I need more end-to-end tests?

@bahmutov

Let the code coverage be your guide

But do not let 100% code coverage be the only goal

Istanbul.js - code instrumentation library

nyc - CLI around Istanbul.js

babel-plugin-istanbul

Code Coverage Demo

@cypress/code-coverage

Cypress.io plugin for saving Istanbul code coverage

import '@cypress/code-coverage/support'

cypress/support/index.js file

module.exports = (on, config) => {
  on('task', require('@cypress/code-coverage/task'))
}

cypress/plugins/index.js file

A single end-to-end test can be very effective at covering a lot of code

The missed lines in the business logic code are targets for more end-to-end tests

Cannot reach the edge case from the end-to-end test

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

cypress/integration/selectors-spec.js

Run unit test from Cypress

and get a perfect 100%

Send code coverage results to external service

https://coveralls.io/github/cypress-io/jsnation-example?branch=master

Beyond code coverage

  • user story coverage

  • UI element coverage

  • state machine coverage

https://slides.com/bahmutov/test-coverage-update

https://slides.com/bahmutov/well-tested-software

Viewport coverage

Does the app work on different screens?

;['macbook-15', 'iphone-6'].forEach(viewport => {
  it(`works on ${viewport}`, () => {
    cy.viewport(viewport)
    cy.visit('/')
    cy.get('.new-todo')
      .type('write code{enter}')
      .type('write tests{enter}')
      .type('deploy{enter}')
    cy.get('.todo').should('have.length', 3)

    cy.get('.todo')
      .first()
      .find('.toggle')
      .check()
    cy.get('.todo')
      .first()
      .should('have.class', 'completed')

    cy.get('.clear-completed').click()
    cy.get('.todo').should('have.length', 2)
  })
})
// import 'todomvc-app-css/index.css'

functional tests still pass!

🚫

Visual testing

∆

=

?

baseline image

new image

visual diff

Visual Testing Challenges

🙁 Hard to test complex app states

     😍 — but not with Cypress!

😵 Performance

😡 Consistent & deterministic rendering

😬 Day-to-day review process & approval workflow.

Lots of options

https://on.cypress.io/plugins#visual-testing

Do-It-Yourself vs 3rd party service

Disclaimer: PhD in Computer Vision and Image Processing

Visual Testing Demo

  • Percy.io plugin and commands

  • Consistent application state

  • GitHub pull request workflow

https://www.cypress.io/blog/2019/04/19/webinar-recording-cypress-io-percy-end-to-end-functional-and-visual-testing-for-the-web/

Pull request workflow

runs first

added async

Cypress GitHub integration is coming soon

https://github.com/cypress-io/cypress/issues/981

Functional tests

Visual testing

Code coverage

Crash reporting

Find friends for your tools

Thank you

Gleb Bahmutov

Cypress.io

@bahmutov

Boston, USA

slides.com/bahmutov/testing-tools-and-their-friends