Gleb Bahmutov, PhD

VP of Engineering, Cypress.io

Test Coverage Update

This.JavaScript - State of Testing

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional / testing

EveryScape

virtual tours

MathWorks

MatLab on the web

Kensho

finance dashboards

these slides

15 people. Atlanta, Philly, Boston, LA, Chicago, NYC

Fast, easy and reliable testing for anything that runs in a browser

Do I need to write more tests?

Code Coverage

  • Instrument the code

  • Run the tests

  • See if we reached every line of code

shows code coverage by labeling code

Coverage is hard

Code coverage

is tricky

const isEmail = (s) =>
  /^\w+@\w+\.\w{3,4}$/.test(s)

// 1 test = 100% code coverage
​(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

Code coverage

vs

Data coverage

mocha -r data-cover spec.js

Do I need to write more end-to-end tests?

I don't think code/data coverage is useful for end-to-end tests

You cannot test every part of your car by driving around

user interface

application code

vendor code

polyfills

there is probably a lot of app code unreachable through the UI alone

beforeEach(() => {
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

What does this test cover?

beforeEach(() => {
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

What does this test cover?

track elements test interacts with, like "type"

Cypress.Commands.overwrite('type', 
  (type, $el, text, options) => {

  rememberSelector($el)

  return type($el, text, options)
})

Track "cy.type" elements

Highlight tested element

Elements NOT covered by the test

beforeEach(() => {
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
    .first().find(':checkbox').check()

  cy.contains('.filters a', 'Active').click()
  cy.url().should('include', 'active')

  cy.contains('.filters a', 'Completed').click()
  cy.url().should('include', 'completed')

  cy.contains('.filters a', 'All').click()
  cy.url().should('include', '#/')
})

Extend test to cover more elements

track "type", "check", "click" elements

The test did not cover "Clear completed" button

Problem: only has noticed the first checkbox, and not all of them

big problem: only "check" this box,

but not "uncheck"

tests

code

coverage

data

coverage

element

coverage

       ?            

Do I need to write more tests?

Gleb Bahmutov, PhD

VP of Engineering

Thank you

@cypress_io

@bahmutov

github.com/cypress-io/cypress ⭐️

www.cypress.io/blog

Test Coverage Update: State of Testing

By Gleb Bahmutov

Test Coverage Update: State of Testing

Code coverage is useful to make sure every line of code with unit tests. But it has a few shortcomings and is mostly useless for end-to-end tests. What else can we do to target out end-to-end tests? In this presentation, I will touch on element coverage that seems to be useful when covering web application tests. In blog post form at https://www.cypress.io/blog/2018/12/20/element-coverage/

  • 1,191
Loading comments...

More from Gleb Bahmutov