How We Introduced Automated Web App Testing in A Large Org

Gleb Bahmutov

Sr Director of Engineering

our planet is in imminent danger

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Agenda

  • The large Co
  • The situation
  • Picking the right testing tool
  • Working with each team
  • The current progress
  • The future

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

The same CTO

Gleb, we need our stuff to work.

  • Web app

  • ReactNative mobile app

  • APIs

  • Special projects

Reconnaissance

Meet people

Every VP / Director

QA team

Web team

ReactNative team

DevOps team

API / Backend team

What are your goals? 

What do we have now?

What is the current process?

What can we do?

Where is everything?

How do I?

Where Things Stand / Stood

Let's debug a failed MABL test

Let's debug a failed Webdriver test

A Very Common Story

  • Web team writes unit tests
  • API services teams write unit tests
  • Very complex system of services
  • QA team tests mostly manually
    • Attempts at automation fail
    • Low confidence overall

Positives 🎉

  • Very modern tech stack
  • GitHub and CircleCI
  • Dev preview environments
  • Global alignment

The Management

  • What do we need to do?
  • What will it give us?

Showing The E2E Tests Benefits

  • Observe the web development

  • Write demo tests

  • Show how they catch the current errors and prevent future regressions

E2E

integration

unit

Really important to users

Really important to developers

E2E

integration

unit

  • logical errors
  • browser quirks
  • style errors
  • database errors
  • backend errors
  • DNS errors
  • API services

The Testing Matrix

Optimize for maximum confidence at minimum effort

The Testing Matrix

Helps refine the testing strategy through iteration

future

sprint 2

sprint 1

We started here

Picking The Right E2E Testing Tool

  • Features
  • Effort

Testing Tools

Effort Simulator:

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

The test runner

No tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Manual tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + Enzyme

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + RTL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Visual Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress with Gleb

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Selenium

Installation

Writing tests

Running tests

Debugging tests

Maintenance

MABL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Joint Decision

Devs working on a feature X write E2E tests for feature X

QA team automates all current manual E2E tests

Timezones!!!

Tokyo is +13 hours from Boston 🤕

The Web Team

I hope these tests are not going to be extra work for us...

Writing these tests and running them as you implement the feature will make your more productive

Typical Cypress E2E test

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

Make Adjustments For Web Team

  • Keep E2E tests in a separate repo

  • Run the changed specs first

  • Share the results widely

Cypress Dashboard: Communication

From Cypress Dashboard to GitHub PR comment via

https://on.cypress.io/github-integration

The QA Team

Help everyone acquire automation skills

  • Demo of E2E testing with Cypress (1 hour)
  • A page with links to multiple Cypress resources (free courses, tutorials, workshops, videos)
  • Dedicated support Slack channel
  • Detailed test examples
  • Practice code along practice workshop (2 hours)
  • Lots of "test writing practice" calendar sessions

Describe Cypress Skills

Scroll to see the list of skills and resources

  • 10 QA people, mostly manual
  • 2.5 writing Cypress automated tests
  • The rest are supporting ReactNative project

There is no instant way to transform manual QA into automation engineers; you have to hire eng with Cy experience

Today

Automation QA Group

1+1

Automation engineer job listing at Mercari US

https://www.mercari.com/careers/?gh_jid=5909531002#grnhse_app

Be a Writer

- https://glebbahmutov.com/blog/cypresss-data-session/
- https://glebbahmutov.com/blog/refactor-using-each/
- https://glebbahmutov.com/blog/dynamic-api-tests-using-cypress-each/
- https://glebbahmutov.com/blog/cypress-and-testrail/
- https://glebbahmutov.com/blog/cypress-skills-ladder/
- https://glebbahmutov.com/blog/faster-ci-feedback-on-circleci/
- https://glebbahmutov.com/blog/request-graphql/
- https://glebbahmutov.com/blog/dynamic-tests-from-fixture/
- https://glebbahmutov.com/blog/skip-cypress-install-on-ci/
- https://glebbahmutov.com/blog/verify-phone-part-one/
- https://glebbahmutov.com/blog/verify-phone-part-two/
- https://glebbahmutov.com/blog/detached/
- https://glebbahmutov.com/blog/visit-blank-page-between-tests/

- https://glebbahmutov.com/blog/burn-tests-on-circle/
- https://glebbahmutov.com/blog/avoid-cypress-pyramid-of-doom/
- https://glebbahmutov.com/blog/control-nextjs-data-during-tests/
- https://glebbahmutov.com/blog/test-feature-flags/
- https://glebbahmutov.com/blog/rn-test-for-navigation/
- https://glebbahmutov.com/blog/test-app-using-the-keyboard/
- https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci/
- https://glebbahmutov.com/blog/cypress-second-tab/
- https://glebbahmutov.com/blog/cypress-real-events/
- https://glebbahmutov.com/blog/sanity-test/
- https://glebbahmutov.com/blog/testing-rn-todo-app/
- https://glebbahmutov.com/blog/stubbing-the-non-configurable/
- https://glebbahmutov.com/blog/testing-react-native-app-using-cypress/

Be Prolific Writer

Be Prolific Writer YouTuber

README

(if any)

Wiki

(if up-to-date)

Slack

(if can find it)

I try to avoid this:

How do I ...

The DevOps Team

The Dev environments are less than dependable

💡 E2E tests can be as quick as the longest-running test

With The DevOps Team

  • Figure out the Dev platform hiccups
  • Make all tests green
  • Drive down flaky tests
    • test retries, coding style, best practices

Make every failed test 🚨 count

Flaky Tests

Drill Into Flaky Test

The API Services Team

  • Web app
  • GraphQL service
  • API services
  • Search service
  • Suggestion service
  • Item pricing service
  • Shipping service
  • Payment service
  • X, Y, Z services

The API Services Team

  • Web app
  • GraphQL service
  • API services
  • Search service
  • Suggestion service
  • Item pricing service
  • Shipping service
  • Payment service
  • X, Y, Z services

By driving the web app like a real user, our E2E tests have found issues with each service

Need To Show Value

Run E2E tests:

  • Web application PRs

  • Web application deploys

  • API PRs

  • API deploys

  • Service X, Y, Z deploys

  • Feature flag changes

  • Database changes

Work In Progress

Current Status

We are writing tests!

QA Team, Web teams

Strategy

  • Wide coverage

  • Prioritize one/two feature areas

  • Port the failing MABL/Selenium tests

Link to manual test plans

describe('User tests', () => {
  it('C111 User can log in', () => {
    cy.wait(1000)
  })

  it('C112 User can log out', () => {
    cy.log('in the last test!')
    cy.wait(1000)
  })
})

Flexibility & Fast Feedback

The Tale Of Two Repos

The Testing Matrix

Helps refine the testing strategy through iteration

future

current work

Future Plans

  • Catch CSS regressions using visual testing
  • Write A11y tests
  • Write component tests
  • Write API tests
  • Test email flows
  • Collect code coverage

It Takes Time & Effort

to change the course

Got to show the E2E test value

Gleb Bahmutov

Sr Director of Engineering

Thank You 👏