How We Introduced Automated Web App Testing in A Large Org

Gleb Bahmutov

Sr Director of Engineering

https://slides.com/bahmutov/testing-large-org

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end and we are close

https://www.ipcc.ch/assessment-report/ar6/

@bahmutov

survival is possible* but we need to act now

change your life
dump banks financing fossil projects
join an organization

@bahmutov

rebellion.global www.wedonthavetime.org

www.sierraclub.org 350.org www.greenpeace.org

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

Agenda

The large Co
The situation
Picking the right testing tool
Working with each team
The current progress
The future

@bahmutov

https://slides.com/bahmutov/testing-large-org

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

The same CTO

Gleb, we need our stuff to work.

Web app
ReactNative mobile app
APIs
Special projects

Reconnaissance

Meet people

Every VP / Director

QA team

Web team

ReactNative team

DevOps team

API / Backend team

What are your goals?

What do we have now?

What is the current process?

What can we do?

Where is everything?

How do I?

Where Things Stand / Stood

Let's debug a failed MABL test

Let's debug a failed Webdriver test

A Very Common Story

Web team writes unit tests
API services teams write unit tests
Very complex system of services
QA team tests mostly manually
- Attempts at automation fail
- Low confidence overall

Positives 🎉

Very modern tech stack
GitHub and CircleCI
Dev preview environments
Global alignment

The Management

What do we need to do?
What will it give us?

Showing The E2E Tests Benefits

Observe the web development
Write demo tests
Show how they catch the current errors and prevent future regressions

E2E

integration

unit

@bahmutov

Really important to users

Really important to developers

E2E

integration

unit

logical errors
browser quirks
style errors
database errors
backend errors
DNS errors
API services

The Testing Matrix

Optimize for maximum confidence at minimum effort

https://slides.com/bahmutov/pyramid-testjsummit (with Roman Sandler)

The Testing Matrix

Helps refine the testing strategy through iteration

future

sprint 2

sprint 1

We started here

https://slides.com/bahmutov/pyramid-testjsummit (with Roman Sandler)

Picking The Right E2E Testing Tool

Features
Effort

Testing Tools

Effort Simulator:

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Joint Decision

Devs working on a feature X write E2E tests for feature X

QA team automates all current manual E2E tests

Timezones!!!

Tokyo is +13 hours from Boston 🤕

The Web Team

I hope these tests are not going to be extra work for us...

Writing these tests and running them as you implement the feature will make your more productive

Typical Cypress E2E test

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

Make Adjustments For Web Team

Keep E2E tests in a separate repo
Run the changed specs first
Share the results widely

https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci/

https://glebbahmutov.com/blog/faster-ci-feedback-on-circleci/

https://on.cypress.io/dashboard-introduction

Cypress Dashboard: Communication

https://on.cypress.io/dashboard-introduction

From Cypress Dashboard to GitHub PR comment via

https://on.cypress.io/github-integration

The QA Team

Help everyone acquire automation skills

Demo of E2E testing with Cypress (1 hour)
A page with links to multiple Cypress resources (free courses, tutorials, workshops, videos)
Dedicated support Slack channel
Detailed test examples
Practice code along practice workshop (2 hours)
Lots of "test writing practice" calendar sessions

Describe Cypress Skills

Scroll to see the list of skills and resources

10 QA people, mostly manual
2.5 writing Cypress automated tests
The rest are supporting ReactNative project

There is no instant way to transform manual QA into automation engineers; you have to hire eng with Cy experience

Today

Automation QA Group

1+1

Automation engineer job listing at Mercari US

https://www.mercari.com/careers/?gh_jid=5909531002#grnhse_app

Be a Writer

- https://glebbahmutov.com/blog/cypresss-data-session/
- https://glebbahmutov.com/blog/refactor-using-each/
- https://glebbahmutov.com/blog/dynamic-api-tests-using-cypress-each/
- https://glebbahmutov.com/blog/cypress-and-testrail/
- https://glebbahmutov.com/blog/cypress-skills-ladder/
- https://glebbahmutov.com/blog/faster-ci-feedback-on-circleci/
- https://glebbahmutov.com/blog/request-graphql/
- https://glebbahmutov.com/blog/dynamic-tests-from-fixture/
- https://glebbahmutov.com/blog/skip-cypress-install-on-ci/
- https://glebbahmutov.com/blog/verify-phone-part-one/
- https://glebbahmutov.com/blog/verify-phone-part-two/
- https://glebbahmutov.com/blog/detached/
- https://glebbahmutov.com/blog/visit-blank-page-between-tests/

- https://glebbahmutov.com/blog/burn-tests-on-circle/
- https://glebbahmutov.com/blog/avoid-cypress-pyramid-of-doom/
- https://glebbahmutov.com/blog/control-nextjs-data-during-tests/
- https://glebbahmutov.com/blog/test-feature-flags/
- https://glebbahmutov.com/blog/rn-test-for-navigation/
- https://glebbahmutov.com/blog/test-app-using-the-keyboard/
- https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci/
- https://glebbahmutov.com/blog/cypress-second-tab/
- https://glebbahmutov.com/blog/cypress-real-events/
- https://glebbahmutov.com/blog/sanity-test/
- https://glebbahmutov.com/blog/testing-rn-todo-app/
- https://glebbahmutov.com/blog/stubbing-the-non-configurable/
- https://glebbahmutov.com/blog/testing-react-native-app-using-cypress/

https://glebbahmutov.com/blog/

Be Prolific Writer

https://cypresstips.substack.com/

Be Prolific Writer YouTuber

https://www.youtube.com/glebbahmutov

https://cypress.tips/search

https://glebbahmutov.com/blog/incremental-post-scraping/

README

(if any)

Wiki

(if up-to-date)

Slack

(if can find it)

I try to avoid this:

How do I ...

The DevOps Team

The Dev environments are less than dependable

💡 E2E tests can be as quick as the longest-running test

With The DevOps Team

Figure out the Dev platform hiccups
Make all tests green
Drive down flaky tests
- test retries, coding style, best practices

Make every failed test 🚨 count

Flaky Tests

https://on.cypress.io/flaky-test-management

Drill Into Flaky Test

The API Services Team

Web app
GraphQL service
API services
Search service
Suggestion service
Item pricing service
Shipping service
Payment service
X, Y, Z services

The API Services Team

Web app
GraphQL service
API services
Search service
Suggestion service
Item pricing service
Shipping service
Payment service
X, Y, Z services

By driving the web app like a real user, our E2E tests have found issues with each service

Need To Show Value

Run E2E tests:

Web application PRs
Web application deploys
API PRs
API deploys
Service X, Y, Z deploys
Feature flag changes
Database changes

Work In Progress

Current Status

We are writing tests!

QA Team, Web teams

Strategy

Wide coverage
Prioritize one/two feature areas
Port the failing MABL/Selenium tests

Link to manual test plans

https://github.com/bahmutov/cypress-testrail-simple

https://glebbahmutov.com/blog/cypress-and-testrail/

describe('User tests', () => {
  it('C111 User can log in', () => {
    cy.wait(1000)
  })

  it('C112 User can log out', () => {
    cy.log('in the last test!')
    cy.wait(1000)
  })
})