Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
The latest updates from
Brian Mann
Gleb Bahmutov
Founder & Creator
VP Engineering
Oct 16, 2018 Webinar
The Agenda
1. Parallelization & Grouping
2. Dashboard paywall
5. Q&A
3. Test retries
4. Cross-browser support
Q & A - use Zoom to ask a question
Cypress test parallelization and grouping
I want my tests to finish quickly
If my CI can allocate 10 machines for me - why can't the tests run 10x faster?
And I want zero config
$ cypress run --record --parallel
Cypress v3.1.0
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
Dashboard
specs
specs
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
a.spec.js
b.spec.js
c.spec.js
...
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
Dashboard
a.spec.js (2s)
b.spec.js (10s)
c.spec.js (5s)
...
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
b.spec.js (10s)
c.spec.js (5s)
a.spec.js (2s)
...
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
🔄 b.spec.js (10s)
c.spec.js (5s)
a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
🔄 b.spec.js (10s)
🔄 c.spec.js (5s)
a.spec.js (2s)
...
c.spec.js
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
c.spec.js
🔄 b.spec.js (10s)
🔄 c.spec.js (5s)
a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
pass / fail
🔄 b.spec.js (10s)
🔄 c.spec.js (5s)
a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
🔄 b.spec.js (10s)
✅ c.spec.js (5s)
a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
🔄 b.spec.js (10s)
✅ c.spec.js (5s)
🔄 a.spec.js (2s)
...
a.spec.js
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
a.spec.js
🔄 b.spec.js (10s)
✅ c.spec.js (5s)
🔄 a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
pass / fail
🔄 b.spec.js (10s)
✅ c.spec.js (5s)
🔄 a.spec.js (2s)
...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
🔄 b.spec.js (10s)
✅ c.spec.js (5s)
✅ a.spec.js (2s)
...
and the process repeats until all specs have completed ...
b.spec.js
Dashboard
Machine #1
$ cypress run --record --parallel
$ cypress run --record --parallel
Machine #2
How parallelization works
cypress-example-kitchensink demo
Zero config
2x-electron:
# tell CircleCI to execute this job on 2 machines simultaneously
parallelism: 2
steps:
- run: npm run e2e:record -- --parallel
Zero config
2x-electron:
# tell CircleCI to execute this job on 5 machines simultaneously
parallelism: 5
steps:
- run: npm run e2e:record -- --parallel
Zero config
Zero config
Balancing act
Split longest running spec
Balancing act
Split longest running spec
Experiment with number of machines
🤖🤖🤖
🤖🤖🤖
🤖🤖🤖
cypress-dashboard parallelization
Text
cypress-dashboard parallelization
| # machines | run duration | time savings |
|---|---|---|
| 1 | 22:50 | ~ |
| 2 | 11:47 | 48% |
| 3 | 7:51 | 65% |
| 4 | 5:56 | 74% |
| 6 | 3:50 | 83% |
| 8 | 3:00 | 87% |
| 10 | 2:19 | 90% |
cypress-dashboard parallelization
10 machines = 10x speed up
| # machines | run duration | time savings |
|---|---|---|
| 1 | 22:50 | ~ |
| 2 | 11:47 | 48% |
| 3 | 7:51 | 65% |
| 4 | 5:56 | 74% |
| 6 | 3:50 | 83% |
| 8 | 3:00 | 87% |
| 10 | 2:19 | 90% |
cypress-dashboard parallelization
10 machines = 10x speed up
Grouping
I want to run a few smoke tests then all tests
I want to run tests, deploy, then run tests against the deployed server
I want to run tests using Electron, then again using Chrome / X
$ cypress run --record --group 1x-electron
$ cypress run --record --group 2x-electron --parallel
Read about grouping
The Dashboard Paywall
Going up October 20th 2018
Test runner remains 100% OSS
Upcoming Dashboard features
Detailed information about every test
Historical information about all tests
Upcoming Dashboard features
Detailed information about every test
Historical information about all tests
Flakiness coefficient
companion to cypress-io/cypress/issues/1313
Test retries, cross-browser support, Q&A
switching slides to Brian Mann ...
Remember to ask a question, we will answer them at the end of the webinar
Cypress Test Parallelization and Grouping
By Gleb Bahmutov
Cypress Test Parallelization and Grouping
Intro to end-to-end test parallelization and grouping with Cypress Dashboard service. Part of Cypress webinar on October 16 2018. Video at https://www.youtube.com/watch?v=FfqD1ExUGlw
