How We Introduced Automated Web App Testing in A Large Organization

Gleb Bahmutov

Sr Director of Engineering

our planet is in imminent danger

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Agenda

  • The large Co
  • The situation
  • Picking the right testing tool
  • Working with each team
  • The current progress
  • The future

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

EveryScape

MathWorks

Kensho

Cypress.io

75 → 20

2000

8 → 100

5 → 50

300 (2000)

The same CTO

Gleb, we need our stuff to work.

  • Web app

  • ReactNative mobile app

  • APIs

  • Special projects

I have kept a list of ideas, bugs, problems related to Cypress and Cypress Dashboard service, the list keeps growing.

A New Appreciation

Reconnaissance

Meet people

Every VP / Director

QA team

Web team

ReactNative team

DevOps team

API / Backend team

What are your goals? 

What do we have now?

What is the current process?

What can we do?

Where is everything?

How do I?

Where Things Stand / Stood

Let's debug a failed MABL test

Let's debug a failed Webdriver test

A Very Common Story

  • Web team writes unit tests
  • API services teams write unit tests
  • Very complex system of services
  • QA team tests mostly manually
    • Attempts at automation fail
    • Low confidence overall

Positives 🎉

  • Very modern tech stack
  • GitHub and CircleCI
  • Dev preview environments
  • Global alignment

The Management

  • What do we need to do?
  • What will it give us?

The original pyramid

Mike Cohn -  Succeeding with Agile (2009)

Showing The E2E Tests Benefits

  • Observe the web development

  • Write demo tests

  • Show how they catch the current errors and prevent future regressions

E2E

integration

unit

Really important to users

Really important to developers

E2E

integration

unit

  • logical errors
  • browser quirks
  • style errors
  • database errors
  • backend errors
  • DNS errors
  • API services

The Testing Matrix

Optimize for maximum confidence at minimum effort

The Testing Matrix

Helps refine the testing strategy through iteration

future

sprint 2

sprint 1

We started here

Picking The Right E2E Testing Tool

  • Features
  • Effort

Testing Tools

Effort Simulator:

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Documentation 

Training

Prev experience

CI Setup

The test runner

No tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

No automated tests

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + Enzyme

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest + RTL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Jest Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Visual Snapshots

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Cypress with Gleb

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Selenium

Installation

Writing tests

Running tests

Debugging tests

Maintenance

MABL

Installation

Writing tests

Running tests

Debugging tests

Maintenance

Joint Decision

Work on a feature X = write E2E tests for feature X

QA team automates all current manual E2E tests

Timezones!!!

Tokyo is +13 hours from Boston 🤕

The Web Team

I hope these tests are not going to be extra work for us...

Writing these tests and running them as you implement the feature will make your more productive

Typical Cypress E2E test

cy.signup(seller)

cy.createListing({
  name: `Macbook one ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 198,
})
cy.createListing({
  name: `Macbook two ${Cypress._.random(1e10)}`,
  description: 'Seller will delete all items',
  price: 199,
})

visitBlankPage()
cy.loginUserUsingAPI(seller)
cy.visitProtectedPage('/mypage/listings/active')
cy.byTestId('Filter', 'Active').should('be.visible').and('contain', '2')
cy.byTestId('ListingRow').should('be.visible').and('have.length', 2)

Pull request template

A typical Cypress test

Make Adjustments For Web Team

  • Keep E2E tests in a separate repo

  • Run the changed specs first

  • Share the results widely

Cypress Dashboard: Communication

From Cypress Dashboard to GitHub PR comment via https://on.cypress.io/github-integration

The QA Team

Help everyone acquire automation skills

  • Demo of E2E testing with Cypress (1 hour)
  • A page with links to multiple Cypress resources (free courses, tutorials, workshops, videos)
  • Dedicated support Slack channel
  • Detailed test examples
  • Practice code along practice workshop (2 hours)
  • Lots of "test writing practice" calendar sessions

Describe Cypress Skills

Scroll to see the list of skills and resources

  • 10 QA people, mostly manual
  • 1.5 writing Cypress automated tests
  • The rest are supporting ReactNative project

There is no instant way to transform manual QA into automation engineers

Hiring An Automation Engineer

  • Likes to work in a fast-paced environment
  • Excellent communication skills
  • 10 years of automation experience
  • 8 years of Cypress experience
  • ....
  • Buzzword, buzzword, buzzword
  • I stopped reading
  1. List all the things we do right now
    1. Provide links to each resource
  2. List all the things we want to do
  3. Join us if you have experience in 1) and want to do 2)

Hiring An Automation Engineer

First Interview

Homework

  1. Take any web application
  2. Write Cypress tests
  3. Send us a link to the repo when ready

Second Interview

About hiring...

Be a Writer

- https://glebbahmutov.com/blog/cypresss-data-session/
- https://glebbahmutov.com/blog/refactor-using-each/
- https://glebbahmutov.com/blog/dynamic-api-tests-using-cypress-each/
- https://glebbahmutov.com/blog/cypress-and-testrail/
- https://glebbahmutov.com/blog/cypress-skills-ladder/
- https://glebbahmutov.com/blog/faster-ci-feedback-on-circleci/
- https://glebbahmutov.com/blog/request-graphql/
- https://glebbahmutov.com/blog/dynamic-tests-from-fixture/
- https://glebbahmutov.com/blog/skip-cypress-install-on-ci/
- https://glebbahmutov.com/blog/verify-phone-part-one/
- https://glebbahmutov.com/blog/verify-phone-part-two/
- https://glebbahmutov.com/blog/detached/
- https://glebbahmutov.com/blog/visit-blank-page-between-tests/

- https://glebbahmutov.com/blog/burn-tests-on-circle/
- https://glebbahmutov.com/blog/avoid-cypress-pyramid-of-doom/
- https://glebbahmutov.com/blog/control-nextjs-data-during-tests/
- https://glebbahmutov.com/blog/test-feature-flags/
- https://glebbahmutov.com/blog/rn-test-for-navigation/
- https://glebbahmutov.com/blog/test-app-using-the-keyboard/
- https://glebbahmutov.com/blog/how-to-keep-cypress-tests-in-another-repo-with-circleci/
- https://glebbahmutov.com/blog/cypress-second-tab/
- https://glebbahmutov.com/blog/cypress-real-events/
- https://glebbahmutov.com/blog/sanity-test/
- https://glebbahmutov.com/blog/testing-rn-todo-app/
- https://glebbahmutov.com/blog/stubbing-the-non-configurable/
- https://glebbahmutov.com/blog/testing-react-native-app-using-cypress/

README

(if any)

Wiki

(if up-to-date)

Slack

(if can find it)

I try to avoid this:

How do I ...

Blog For Yourself

Your coworkers will thank you

The DevOps Team

The Dev environments are less than dependable

💡 E2E tests can be as quick as the longest-running test

With The DevOps Team

  • Figure out the Dev platform hiccups
  • Make all tests green
  • Drive down flaky tests
    • test retries, coding style, best practices

Make every failed test 🚨 count

Flaky Tests

The API Services Team

  • Web app
  • GraphQL service
  • API services
  • Search service
  • Suggestion service
  • Item pricing service
  • Shipping service
  • Payment service
  • X, Y, Z services

The API Services Team

  • Web app
  • GraphQL service
  • API services
  • Search service
  • Suggestion service
  • Item pricing service
  • Shipping service
  • Payment service
  • X, Y, Z services

By driving the web app like a real user, our E2E tests have found issues with each service

Need To Show Value

Run E2E tests:

  • Web application PRs

  • Web application deploys

  • API PRs

  • API deploys

  • Service X, Y, Z deploys

  • Feature flag changes

  • Database changes

Current Status

We are writing tests!

QA Team, Web teams

Strategy

  • Wide coverage

  • Prioritize one/two feature areas

  • Port the failing MABL/Selenium tests

Link to manual test plans

describe('User tests', () => {
  it('C111 User can log in', () => {
    cy.wait(1000)
  })

  it('C112 User can log out', () => {
    cy.log('in the last test!')
    cy.wait(1000)
  })
})

Things To Watch Out For

This feature (API, element selector, CI job, ...) is only needed for your tests...

This feature (API, element selector, CI job, ...) is necessary to have confidence in our application and services

Things To Watch Out For

If your team needs it, you do it. Be bold. Be fullstack.

Can I speak to your manager?

Things To Watch Out For

It is on our backlog already. It has been there for months.

Let's schedule a regular status meetings (with CTO) until it gets done.

The Testing Matrix

Helps refine the testing strategy through iteration

future

current work

Future Plans

  • Catch CSS regressions using visual testing
  • Write A11y tests
  • Write component tests
  • Write API tests
  • Have every service tested automatically via E2E tests
  • Collect code coverage

It Takes Time & Effort

to change the course

Gleb Bahmutov

Sr Director of Engineering

Thank You 👏

How We Introduced Automated Web App Testing in A Large Organization

By Gleb Bahmutov

How We Introduced Automated Web App Testing in A Large Organization

Many organizations struggle with building a reliable set of automated tests for their web applications. Yet a set of manual test plans is not an answer; testing without automation is slow, gives low confidence in the software, and the test results provide little actionable feedback to the developers. In this presentation, I describe how a large company has built a comprehensive automated set of end-to-end test suites after two previous unsuccessful attempts. I will show how to build trust between the QA engineers and front-end developers, how to convince other stakeholders that investing time in testing is important, and how to balance the different types of tests to cover both the happy paths and the edge cases.

  • 2,352