End to End Testing in the Future
Gleb Bahmutov
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional
EveryScape
virtual tours
MathWorks
MatLab on the web
Kensho
finance dashboards
11 people. Atlanta, Philly, Boston, LA
Fast, easy and reliable testing for anything that runs in a browser
I have Cypress.io t-shirts (a few)
-
Testing pyramid
-
E2E testing: the bad parts
-
Cypress.io demo
-
The future of testing
E2E
integration
unit
Smallest pieces
Testing Pyramid △
Unit tests pass...
E2E
integration
unit
Component
putting things together
E2E
integration
unit
Component
Enzyme full rendering
E2E
integration
unit
Website / app
Will Klein
Let's test!
- Open and control real browser
Your tests
Selenium
Driver
FF, Chrome , IE
Jennifer Shehane @jennifershehane
Let's test!
Your tests
FF, Chrome , IE
slow, flaky, unreliable tests
WebDriver = JSON over HTTP
Let's test!
Your tests
FF, Chrome , IE
Let's test!
Your tests
Headless browser
It is working ... I think
Let's test!
Your tests
Chrome Puppeteer
I need more parts
GUI showing tests
Assertion library
Automatic retries
Video recording and screenshots
Mocking and stubbing
Network control
CI support
...
$ npm install -D cypress
- Win / Mac / Linux
- MIT License
- OSS https://github.com/cypress-io/cypress
- https://on.cypress.io/introduction-to-cypress
it('opens the page', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.should('be.visible')
})it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})Cypress demo
- Opening a page
- Typical test
- Failing test
- API and documentation
- Recording tests
- CI setup
To learn more:
Egghead.io Cypress Course
Free course (same Egghead.io author!)
Cypress documentation
How does Cypress make 💵
Cypress Dashboard
See, share, and fix failing tests.
Cypress Dashboard
- Is a paid Saas complement to Cypress app
- Perfect place to store test results, screenshots and videos
- Helps find why and when a test started failing
Testing of the Future
Component testing
Smart snapshots
Tests as documentation
Load balancing
Testing of the Future
Component testing
Smart snapshots
Tests as documentation
Load balancing
E2E
integration
unit
E2E
integration
unit
Need:
- real browser (DOM, storage, ...)
- state cleanup between tests
- stubbing server
E2E
integration
unit
Need:
- real browser (DOM, storage, ...)
- state cleanup between tests
- stubbing server
WAIT A MINUTE!
E2E
integration
unit
Cypress
it('logs user in', () => {
cy.visit('page.com')
cy.get('#login').click()
})E2E
integration
unit
Cypress for Integration
import LoginComponent from './src/login'it('logs user in', () => {
mount(LoginComponent)
cy.get('#login').click()
})import mount from 'cypress-<X>-unit-test'E2E
integration
unit
import LoginComponent from './src/login'it('logs user in', () => {
mount(LoginComponent)
cy.get('#login').click()
})import mount from 'cypress-<X>-unit-test'
Cypress for Integration
E2E
integration
unit
import LoginComponent from './src/login'it('logs user in', () => {
mount(LoginComponent)
cy.get('#login').click()
})import mount from 'cypress-<X>-unit-test'storybook.js.org
Cypress for Integration
Interact, inspect, use
Use end-to-end test runner to run your component and unit tests
Testing of the Future
Component testing
Smart snapshots
Tests as documentation
Load balancing
Why ?????
What has changed?
𝚫
(change in mathematics is usually written using Greek letter "delta")
✅ build #999
🛑 build #1000
compare current value with a good value
expect(value).toMatchSnapshot()
Received value does not match stored snapshot 1.
- 3
+ 21Showing ✅ and 🛑 values works for simple values
For everything else: show diff
Object snapshots
it('compares objects', () => {
const o = {
inner: {
a: 10,
b: 20
},
foo: 'foo (changed)',
newProperty: 'here'
}
snapshot(o)
})changed value
new value
Text snapshots
it('compares multi line strings', () => {
snapshot(`
this is a line
and a second line (changed)
with number 42
`)
})changed line
Screenshot diffing
it('looks the same', () => {
expect(cy.screenshot()).toMatchSnapshot()
})[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FF, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...Screenshot diffing
it('looks the same', () => {
expect(cy.screenshot()).toMatchSnapshot()
})[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...Screenshot diffing
[FF, FF, FF] [FF, FF, FF] [FF, FF, FF]
[FF, FE, FF] [FF, FF, FF] [FF, FE, FF]
[FF, FE, FF] [FF, FE, FF] [FF, FE, FF]
[FF, FF, FF] [FF, FE, FF] [FF, FF, FF]
...Shows us WHAT has changed
Shows us WHAT has changed
Welcome!
screenshot difference
DOM
difference
Rewind back
Welcome!
screenshot difference
.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}+ <div>Welcome!</div>style
dom
Rewind back
Welcome!
screenshot difference
.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}+ <div>Welcome!</div>DOM diff tells HOW page has changed
style
dom
.logo {
- background-color: #ffffff;
+ background-color: #fefefe;
}+ <div>Welcome!</div>Application behavior difference
Rewind back some more
style
dom
Passing test
Failing test
HTML, styles, code, cookies, storage, API calls, user interaction
test timeline
pixel + DOM difference
API call returned different greeting text
successful test run
failed test run
test timeline
Test history diff explains WHY the page has changed
successful test run
failed test run
Record Everything?
HAR on steroids
network + DOM + custom
Error, stack trace, screenshot diff
Video of the accident
Car telemetry
Driver's bio data
Find what really happened
See: The Testing Pyramid @AssertJS
Testing of the Future
Component testing
Smart snapshots
Tests as documentation
Load balancing
Tests are for everyone
Use Case Coverage
- User logs in
- User changes primary email
- User can find an item
- ...
describe('User', () => {
it('logs in', () => {
cy.visit(...)
...
})
it('changes primary email', () => {
...
})
it('can find an item', () => {
...
})
})Use Case Coverage
- User logs in
- User changes primary email
- User can find an item
- ...
Tests as documentation replace estimates (guesses) with repeated demos (tests)
Testing of the Future
Component testing
Smart snapshots
Tests as documentation
Load balancing
I don't want to wait 15 minutes
I can't wait 15 minutes
Automatic Test Load Balancing
cypress run --record --parallelversion: 2
jobs:
test:
parallelism: 4CircleCI v2 example
Automatic Test Load Balancing
cypress run --record --parallela-spec.js
b-spec.js
c-spec.js
d-spec.js
e-spec.js
f-spec.js
...
...
...
...
Done!
Automatic Test Load Balancing
- Run specs alphabetically
- Short specs first
- Run previously failed specs first
- Run specs affected by the code first
- ...
Automatic test load balancing makes end-to-end testing fast
End to end testing was hard
If the right tool does not exist - make it
Future of testing is bright
Component testing - ✅
Smart snapshots - in planning
Tests as documentation - PoC
Load balancing - ⏰
Cypress.io