Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
End-to-end testing is hard -
but it doesn't have to be
Gleb Bahmutov, PhD
VP of Engineering
About me:
that is where these slides are
12 people. Atlanta, Philly, Boston, LA, Chicago
Fast, easy and reliable testing for anything that runs in a browser
Cypress.io presentation at ReactiveConf 2016 https://www.youtube.com/watch?v=lK_ihqnQQEM
2016 vs 2018
October 2017: Cypress goes open source
- MIT license
- no feature limitations
- high-quality docs
2016 vs 2018
2016 vs 2018
Going beyond single test run
- Record test artifacts from any CI
- Test parallelization, grouping, insights
- Free for public projects
In this talk:
-
Command retries
-
Test against the browser, not code
-
Browser <-> Node
-
Declarative syntax
-
Customizing Cypress
Command retries
it('adds items', function () {
cy.get('.new-todo')
.type('todo A{enter}')
.type('todo B{enter}')
.type('todo C{enter}')
.type('todo D{enter}')
cy.get('.todo-list li').should('have.length', 2)
})
Let's fail on purpose
it('adds items', function () {
cy.get('.new-todo')
.type('todo A{enter}')
.type('todo B{enter}')
.type('todo C{enter}')
.type('todo D{enter}')
cy.get('.todo-list li').should('have.length', 2)
})
Command + assertion
command
assertion
it('adds items', function () {
cy.get('.new-todo')
.type('todo A{enter}')
.type('todo B{enter}')
.type('todo C{enter}')
.type('todo D{enter}')
cy.get('.todo-list li', {timeout: 1000000})
.should('have.length', 2)
})
Retry longer
helping live web application during the test
Which command is the last one?
cy.get('.todo-list')
<ul class=".todo-list">
...
</ul>
cy.get('.todo-list').find('li')
<ul class=".todo-list">
<li>...</li>
<li>...</li>
...
</ul>
cy.get('.todo-list').find('li').should('have.length', 2)
<ul class=".todo-list">
<li>...</li>
<li>...</li>
...
</ul>
cy.get('.todo-list').find('li').should('have.length', 2)
last command
assertion
<ul class=".todo-list">
<li>...</li>
<li>...</li>
...
</ul>
cy.get('.todo-list').find('li').should('have.length', 2)
last command
assertion
<ul class=".todo-list">
<li>...</li>
<li>...</li>
...
</ul>
Only the last command is retried
cy.get('.todo-list').find('li').should('have.length', 2)
last command
assertion
<ul class=".todo-list">
<li>...</li>
<li>...</li>
...
</ul>
Only the last command is retried
cy.get('.todo-list').find('li').should('have.length', 2)
last command
assertion
cy.get('.todo-list li').should('have.length', 2)
last command
assertion
Only the last command is retried
cy.window()
.its('app').its('$store').its('items')
.should('have.length', 2)
cy.window()
.its('app.$store.items')
.should('have.length', 2)
Maybe "$store" does not exist yet, or the "app" has not yet!
if (window.Cypress) {
window.app = app
}app code
test code
cy.window()
.then(...)
.should('have.length', 2)
const myLogic = () => {
// custom retry logic
return new Promise(...)
}
cy.window()
.then(myLogic)
.should('have.length', 2)
.then() command is NOT retried
const throwDice = () => Cypress._.random(1, 6, false)
// promise-returning functions are more realistic
// in the browser world
const getDiceToBe4 = () => throwDice() === 4
? Cypress.Promise.resolve(4)
: Cypress.Promise.reject(new Error('no dice'))
Bring your own retry logic
// use promise-retry to re-execute until resolves
const promiseRetry = require('promise-retry')
const myLogic = () => promiseRetry((retry, number) => {
return getDiceToBe4().catch(retry)
}, { factor: 1, minTimeout: 100 })Bring your own retry logic
it('retries inside .then', function () {
cy.then(myLogic).should('equal', 4)
})import Convergence from '@bigtest/convergence'
it('converges inside .then', function () {
const throwDice = () =>
Cypress._.random(1, 6, false)
const myLogic = () => {
return new Convergence()
.when(() => throwDice() === 4)
.run()
.then(() => 4) // have to return a value
}
cy.then(myLogic).should('equal', 4)
})
Bring retry logic from other testing tools 😃
Command retries
End-to-end tests must deal well with the unpredictable nature of the web.
But how does it FEEL?
When I test with Cypress
My test context is well isolated from the app's context
Gloves when not in use are always funny
DOM
Network
storage
DOM
Network
storage
framework-agnostic
implementation-agnostic
If you can write E2E tests in a framework-agnostic way
You can replace framework X with Y
(without breaking things)
What about other sides like file system or databases?
Node
Cy backend
cy.task(name, ...args)
on('task', {
name: (...args) => ...
})
Jumping from browser to backend context in tests
result
plugins file
it('finds record in the database', () => {
// random text to avoid confusion
const id = Cypress._.random(1, 1e6)
const title = `todo ${id}`
cy.get('.new-todo').type(`${title}{enter}`)
})Observe Database Effect
runs in the browser
drive via DOM
it('finds record in the database', () => {
// random text to avoid confusion
const id = Cypress._.random(1, 1e6)
const title = `todo ${id}`
cy.get('.new-todo').type(`${title}{enter}`)
cy.task('hasSavedRecord', title).should('equal', true)
})Observe Database Effect
runs in the browser
const hasRecordAsync = (title, ms) => {
// use promise-retry or convergence
...
}
module.exports = (on, config) => {
on('task', {
hasSavedRecord (title, ms = 3000) {
return hasRecordAsync(title, ms)
}
})
}runs in Node in cypress/plugins/index.js
Observe Database Effect
task completes as soon as the server gets POST from the app and saves record to DB
task checks for wrong title and eventually times out
Node to Browser actions are hard
Node
Browser
WebDriver.execute
browser.execute(script[,argument1,...,argumentN]);Cy.task
cy.task(name, ...args)cy.task cy.exec cy.request
(and you only send data, not code)
Browser to Node is easy
it('changes the URL when "awesome" is clicked', () => {
cy.visit('/my/resource/path')
cy.get('.awesome-selector')
.click()
cy.url()
.should('include',
'/my/resource/path#awesomeness')
})Declarative Syntax
there are no async / awaits or promise chains
Tests should read naturally
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
await browser.close();
})();Puppeteer
test('My Test', async t => {
await t
.setNativeDialogHandler(() => true)
.click('#populate')
.click('#submit-button');
const location = await t.eval(() => window.location);
await t.expect(location.pathname)
.eql('/testcafe/example/thank-you.html');
});TestCafe
it('changes the URL when "awesome" is clicked', () => {
const user = cy
user.visit('/my/resource/path')
user.get('.awesome-selector')
.click()
user.url()
.should('include',
'/my/resource/path#awesomeness')
})Cypress is like a real user
Kent C Dodds https://testingjavascript.com/
it('changes the URL when "awesome" is clicked', () => {
cy.visit('/my/resource/path')
cy.get('.awesome-selector')
.click()
cy.url()
.should('include',
'/my/resource/path#awesomeness')
})Cypress: all commands are in a queue
visit
get
click
it('changes the URL when "awesome" is clicked', () => {
cy.visit('/my/resource/path')
cy.get('.awesome-selector')
.click()
cy.url()
.should('include',
'/my/resource/path#awesomeness')
})Cypress: all commands are in a queue
visit
get
click
url
should
Single Command Queue
visit
get
click
url
should
Like a SINGLE user driving the browser
Single Command Queue
visit
get
click
url
should
Deterministic and repeatable
Single Command Queue
visit
get
click
url
should
Lazy
Single Command Queue
visit
get
click
url
should
Lazy
Single Command Queue
visit
get
click
url
should
Lazy
Single Command Queue
visit
get
click
url
should
Lazy
Single Command Queue
visit
get
click
url
should
Lazy
Single Command Queue
visit
get
click
url
should
Commands can be retried (unlike promises)
it('changes the URL when "awesome" is clicked', () => {
cy.visit('/my/resource/path')
cy.get('.awesome-selector')
.click()
const url = await cy.url()
url.should('include',
'/my/resource/path#awesomeness')
})NO!
Promises and async / await are eager single try constructs that do not work (well enough) for E2E tests
visit
get
click
url
should
Test starts
water starts flowing
can water reach finish?
visit
get
click
url
should
Test starts
events starts flowing
can events reach subscribe?
Reactive stream
range(0, 10).pipe(
filter(x => x % 2 === 0),
map(x => x + x),
scan((acc, x) => acc + x, 0)
).subscribe(x =>
console.log(x))visit
get
click
url
should
writing Cypress test is like writing a single reactive stream
Cypress vs X
People testing with Selenium / WebDriver
Cypress users?
No E2E tests
Make The Test Runner Yours
Huge list at https://github.com/bnb/awesome-hyper
before(() => {
console.log('parent.window.document.body is',
parent.window.document.body)
})before(() => {
const $head = Cypress.$(parent.window.document.head)
const css = '...' // new style
$head.append(`<style
type="text/css" id="cypress-dark">
${css}
</style>`)
})your test code
your test code
End-to-end testing is hard -
but it doesn't have to be
Gleb Bahmutov, PhD
VP of Engineering, Cypress.io
Thank you 👏
End-to-end testing is hard - but it doesn't have to be
By Gleb Bahmutov
End-to-end testing is hard - but it doesn't have to be
Cypress.io is an open source free end-to-end test runner that runs your web application inside an iframe in a browser it can closely control. This architecture completely changes how your tests behave (hint: no flake), what you can test (pretty much everything), and why you write more e2e tests (because you love writing them). In this talk, I will give a very brief intro to Cypress itself but mostly will concentrate on how it changes the entire testing experience. Cypress users can test web applications by observing just the external behavior (DOM, network calls, storage) or can reach deep inside the application's code, and I will explain the trade-offs. Finally, I will discuss why Cypress tests do not use "async / await" keywords, and instead look like declarative reactive streams. Video https://www.youtube.com/watch?v=swpz0H0u13k from ReactiveConf 2018
