How We Introduced Automated Web App Testing in A Large Organization
Gleb Bahmutov
Sr Director of Engineering
our planet is in imminent danger
https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end and we are close
survival is possible* but we need to act now
- change your life
- dump banks financing fossil projects
- join an organization
Speaker: Gleb Bahmutov PhD
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
Gleb Bahmutov
Sr Director of Engineering
Agenda
- The large Co
- The situation
- Picking the right testing tool
- Working with each team
- The current progress
- The future
EveryScape
MathWorks
Kensho
Cypress.io
75 → 20
2000
8 → 100
5 → 50
300 (2000)
EveryScape
MathWorks
Kensho
Cypress.io
75 → 20
2000
8 → 100
5 → 50
300 (2000)
The same CTO
Gleb, we need our stuff to work.
-
Web app
-
ReactNative mobile app
-
APIs
-
Special projects
Reconnaissance
Meet people
Every VP / Director
QA team
Web team
ReactNative team
DevOps team
API / Backend team
What are your goals?
What do we have now?
What is the current process?
What can we do?
Where is everything?
How do I?
Where Things Stand / Stood
Let's debug a failed MABL test
Let's debug a failed Webdriver test
A Very Common Story
- Web team writes unit tests
- API services teams write unit tests
- Very complex system of services
- QA team tests mostly manually
- Attempts at automation fail
- Low confidence overall
Positives 🎉
- Very modern tech stack
- GitHub and CircleCI
- Dev preview environments
- Global alignment
The Management
- What do we need to do?
- What will it give us?
Showing The E2E Tests Benefits
-
Observe the web development
-
Write demo tests
-
Show how they catch the current errors and prevent future regressions
E2E
integration
unit
Really important to users
Really important to developers
E2E
integration
unit
- logical errors
- browser quirks
- style errors
- database errors
- backend errors
- DNS errors
- API services
The Testing Matrix
Optimize for maximum confidence at minimum effort
https://slides.com/bahmutov/pyramid-testjsummit (with Roman Sandler)
The Testing Matrix
Helps refine the testing strategy through iteration
future
sprint 2
sprint 1
We started here
https://slides.com/bahmutov/pyramid-testjsummit (with Roman Sandler)
Picking The Right E2E Testing Tool
- Features
- Effort
Testing Tools
Effort Simulator:
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
CI Setup
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Documentation
Training
Prev experience
CI Setup
The test runner
No tests
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Manual tests
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest + Enzyme
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest + RTL
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Jest Snapshots
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Visual Snapshots
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Cypress
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Cypress with Gleb
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Selenium
Installation
Writing tests
Running tests
Debugging tests
Maintenance
MABL
Installation
Writing tests
Running tests
Debugging tests
Maintenance
Joint Decision
Work on a feature X = write E2E tests for feature X
QA team automates all current manual E2E tests
Timezones!!!
Tokyo is +13 hours from Boston 🤕
The Web Team
I hope these tests are not going to be extra work for us...
Writing these tests and running them as you implement the feature will make your more productive
Typical Cypress E2E test
cy.signup(seller)
cy.createListing({
name: `Macbook one ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 198,
})
cy.createListing({
name: `Macbook two ${Cypress._.random(1e10)}`,
description: 'Seller will delete all items',
price: 199,
})
visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)
Pull request template
A typical Cypress test
Make Adjustments For Web Team
-
Keep E2E tests in a separate repo
-
Run the changed specs first
-
Share the results widely
Cypress Dashboard: Communication
From Cypress Dashboard to GitHub PR comment via
The QA Team
Help everyone acquire automation skills
- Demo of E2E testing with Cypress (1 hour)
- A page with links to multiple Cypress resources (free courses, tutorials, workshops, videos)
- Dedicated support Slack channel
- Detailed test examples
- Practice code along practice workshop (2 hours)
- Lots of "test writing practice" calendar sessions
Describe Cypress Skills
Scroll to see the list of skills and resources
- 10 QA people, mostly manual
- 2.5 writing Cypress automated tests
- The rest are supporting ReactNative project
There is no instant way to transform manual QA into automation engineers: hire eng with Cy experience
Today
Automation QA Group
1+1
Be a Writer
- https://glebbahmutov.com/blog/cypresss-data-session/
- https://glebbahmutov.com/blog/refactor-using-each/
- https://glebbahmutov.com/blog/dynamic-api-tests-using-cypress-each/
- https://glebbahmutov.com/blog/cypress-and-testrail/
- https://glebbahmutov.com/blog/cypress-skills-ladder/
- https://glebbahmutov.com/blog/faster-ci-feedback-on-circleci/
- https://glebbahmutov.com/blog/request-graphql/
- https://glebbahmutov.com/blog/dynamic-tests-from-fixture/
- https://glebbahmutov.com/blog/skip-cypress-install-on-ci/
- https://glebbahmutov.com/blog/verify-phone-part-one/
- https://glebbahmutov.com/blog/verify-phone-part-two/
- https://glebbahmutov.com/blog/detached/
- https://glebbahmutov.com/blog/visit-blank-page-between-tests/
- https://glebbahmutov.com/blog/burn-tests-on-circle/
- https://glebbahmutov.com/blog/avoid-cypress-pyramid-of-doom/
- https://glebbahmutov.com/blog/control-nextjs-data-during-tests/
- https://glebbahmutov.com/blog/test-feature-flags/
- https://glebbahmutov.com/blog/rn-test-for-navigation/
- https://glebbahmutov.com/blog/test-app-using-the-keyboard/
- https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci/
- https://glebbahmutov.com/blog/cypress-second-tab/
- https://glebbahmutov.com/blog/cypress-real-events/
- https://glebbahmutov.com/blog/sanity-test/
- https://glebbahmutov.com/blog/testing-rn-todo-app/
- https://glebbahmutov.com/blog/stubbing-the-non-configurable/
- https://glebbahmutov.com/blog/testing-react-native-app-using-cypress/
Be Prolific Writer
Be Prolific Writer YouTuber
README
(if any)
Wiki
(if up-to-date)
Slack
(if can find it)
I try to avoid this:
How do I ...
The DevOps Team
The Dev environments are less than dependable
💡 E2E tests can be as quick as the longest-running test
With The DevOps Team
- Figure out the Dev platform hiccups
- Make all tests green
- Drive down flaky tests
- test retries, coding style, best practices
Make every failed test 🚨 count
Flaky Tests
Drill Into Flaky Test
The API Services Team
- Web app
- GraphQL service
- API services
- Search service
- Suggestion service
- Item pricing service
- Shipping service
- Payment service
- X, Y, Z services
The API Services Team
- Web app
- GraphQL service
- API services
- Search service
- Suggestion service
- Item pricing service
- Shipping service
- Payment service
- X, Y, Z services
By driving the web app like a real user, our E2E tests have found issues with each service
Need To Show Value
Run E2E tests:
-
Web application PRs
-
Web application deploys
-
API PRs
-
API deploys
-
Service X, Y, Z deploys
-
Feature flag changes
-
Database changes
Work In Progress
Current Status
We are writing tests!
QA Team, Web teams
Strategy
-
Wide coverage
-
Prioritize one/two feature areas
-
Port the failing MABL/Selenium tests
Link to manual test plans
describe('User tests', () => {
it('C111 User can log in', () => {
cy.wait(1000)
})
it('C112 User can log out', () => {
cy.log('in the last test!')
cy.wait(1000)
})
})Things To Watch Out For
This feature (API, element selector, CI job, ...) is only needed for your tests...
This feature (API, element selector, CI job, ...) is necessary to have confidence in our application and services
Things To Watch Out For
If your team needs it, you do it. Be bold. Be fullstack.
Can I speak to your manager?
Things To Watch Out For
It is on our backlog already. It has been there for months.
Let's schedule a regular status meetings (with CTO) until it gets done.
The Testing Matrix
Helps refine the testing strategy through iteration
future
current work
Future Plans
- Catch CSS regressions using visual testing
- Write A11y tests
- Write component tests
- Write API tests
- Have every service tested automatically via E2E tests
- Collect code coverage
It Takes Time & Effort
to change the course
Gleb Bahmutov
Sr Director of Engineering
Thank You 👏