Does This Look Ok To You, Computer?

Gleb Bahmutov

Join others to fight the climate crisis

Agenda

  • End-to-end testing

  • Assertion types

  • Visual AI assertions

  • Fail or not

  • Future is 🔆

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

🌎 🔥 350.org 🌎 🔥 citizensclimatelobby.org 🌎 🔥

Gleb Bahmutov

Sr Director of Engineering

If you like it,

Put a 💍 test on it...

Gleb Beyonce Bahmutov

A typical Mercari US Cypress E2E test

Repeatable

Deterministic

Consistent

Reliable

Single user flow

End-to-end Tests

What's there?

End-to-end Tests

  • Go roam freely

  • Notice weird things

  • Investigate

Exploratory Testing

  • Go roam freely

  • Notice weird things

  • Record results

Exploratory Testing

Find unknown bugs, edge cases

Exploratory

Verify each user flow

E2E

Humans

Automation

Time-boxed when available

Pull requests, cron jobs

Find unknown bugs, edge cases

Exploratory

Verify each user flow

E2E

Humans

Automation

Time-boxed when available

Pull requests, cron jobs

Great!

expensive

Humans are human

Mercari Assertion Pyramid

Functional

A11y

Visual diffing

Visual AI

Functional

cy.get('#price')
  .should('be.visible')
  .and('have.text', '$100.00')

Mercari Assertion Pyramid

Functional

A11y

A11y

// once we are done with visual diffing, check accessibility
// https://github.com/sclavijosuero/wick-a11y
cy.checkAccessibility(undefined, {
  includedImpacts: [],
  onlyWarnImpacts: ['critical', 'serious'],
})

Visual diffing

cy.byTestId('ItemDetailsSellerUserName', userName).invoke(
  'text',
  '@user12345678',
)
cy.byTestId('ItemDetailsPosted', /^\d\d\/\d\d\/\d\d$/)
// replace the date with a fixed value
  .invoke('text', '12/31/24')

cy.imageDiff('listing-guest-page', {
  ...VISUAL_TESTING.DESKTOP_COMPARISON_OPTIONS,
  ignoreElements: tid('ItemDetailSimilarItemFeed'),
  diffPercentage: 0.03,
})

Gold image

Visual diffing problems

cy.byTestId('ItemDetailsSellerUserName', userName).invoke(
  'text',
  '@user12345678',
)
cy.byTestId('ItemDetailsPosted', /^\d\d\/\d\d\/\d\d$/)
// replace the date with a fixed value
  .invoke('text', '12/31/24')

cy.imageDiff('listing-guest-page', {
  ...VISUAL_TESTING.DESKTOP_COMPARISON_OPTIONS,
  ignoreElements: tid('ItemDetailSimilarItemFeed'),
  diffPercentage: 0.03,
})

dealing with data

storing and updating gold images

Mercari Assertion Pyramid

Functional

A11y

Visual diffing

Visual AI

ChatGPT.com

Visual AI

ChatGPT.com

Spot on

ignore

Visual AI

cy.location('pathname').should('equal', '/mypage/profile/')
cy.byTestId('ProfilePartial').should('be.visible')
cy.checkViewportUsingAi({
  label: 'user profile edit',
})

Visual AI

const { screenshotName, specName, fullTestName } = getScreenshotName(label)
let filename: string

cy.wrap($el, { log: false })
  .screenshot(screenshotName, {
  	// use extra padding, otherwise AI analysis always complains
  	// that the element is too close to the edge of the image
  	padding,
  	overwrite: true,
  	onAfterScreenshot($el, props) {
    	filename = props.path
  	},
  })
  .then(() => {
    // call Node.js code
    doImageAnalysis(...)
  })

inside cy.checkViewportUsingAi

const instructions = `
	look at the provided image of the web page.
	find any layout or position visual issues that might be present in the web page.
	do not list issues related to whitespace unless they are significant.
	for each issue, label its severity as either "critical", "major", or "minor".

    ${
    	additionalInstructions
    	? `Also, consider the following additional instructions: ${additionalInstructions}`
    	: ''
    }
`

const input = [
  {
    role: 'user',
    content: [
      {
        type: 'input_image',
        image_url: imageDataUri, // screenshot base64 encoding
      },
    ],
  },
]
const answer = await callLiteLLM(instructions, input)

Currently using gpt-5.5

Example: Shipping options

cy.byTestId('DeliveryMethod').should('be.visible')
cy.byTestId('DeliveryMethod').checkUsingAi({
  label: 'Delivery method initial',
  minErrorLevel: 'minor',
  mode: 'warn',
  padding: 5,
})

functional assertion first

visual assertion second

.checkUsingAi({
  label: 'Delivery method initial',
  minErrorLevel: 'minor',
  mode: 'warn',
  padding: 5,
})

Human-readable label for the component or page

.checkUsingAi({
  label: 'Delivery method initial',
  minErrorLevel: 'minor',
  mode: 'warn',
  padding: 5,
})
  • Minor: whitespace issue
  • Major: overlapping buttons
  • Critical: button is cut off
.checkUsingAi({
  label: 'Delivery method initial',
  minErrorLevel: 'minor',
  mode: 'warn',
  padding: 5,
})
  • mode is "warn" (default) or "error"
.checkUsingAi({
  label: 'Delivery method initial',
  minErrorLevel: 'minor',
  mode: 'warn',
  padding: 5,
})

Minor error: an extra line on the top of the component

Example: Shipping options 1/2

The radio button for the “Ship on your own” option is vertically misaligned; it sits noticeably lower than the option title and is inconsistent with the selected radio button above, which aligns with its title.

🤖:

Example: Shipping options 2/2

The first “Shipping label” field uses a right-facing chevron while the adjacent select field uses a down caret, creating an inconsistent control appearance/alignment for similar dropdown-style inputs.

🤖:

Example: Download app page

Example: Download app page

Example: Download app page

major: Missing space between "Mercari balance." and "You've', causing the text to run together as "balance.You've".

🤖:

What Should We Check?

Pick the most important screens for the user flow

  • sign up

  • login

  • create a listing

  • listing

  • checkout

What Should We Check?

💡 check component when empty and with data

SellPage.visit()
cy.byTestId('DeliveryMethod').checkUsingAi({
  label: 'Delivery method initial',
})
SellPage.addCategory()
hasShippingRecommendationApplied()
cy.byTestId('DeliveryMethod').checkUsingAi({
  label: 'Delivery method picked',
})

Common Problems

Solution: remove elements

cy.get('header').invoke('remove')
cy.byTestId('CartPageWrapper').checkUsingAi({
  label: 'cart page',
  padding: 0,
  mode: 'warn',
})

Common Problems

Solution: hide the scrollbars

cy.hideScrollbars()
cy.checkViewportUsingAi({
  label: 'user profile edit',
})

Common Problems

Solution: individual addition instructions

cy.checkViewportUsingAi({
  label: 'listing with a cart',
  instructions: stripIndent`
	The chat overlay column on the right sits on top of the product listing.
	It is expected that it will cover some of the listing details.
	Inside the chat overlay, the quick response chips are expected to
	overflow the chat column horizontally, the user can scroll to find the chip.
  `,
})

Mercari Assertion Pyramid

Functional

A11y

Visual diffing

Visual AI

Mercari Assertion Pyramid

Functional

A11y

Visual diffing

Visual AI

10,000s

10s

20

Assertions: fail or not?

Functional

A11y

Visual diffing

Visual AI

Must pass

tolerance %

human followup

Visual AI: Run Only On CI

Visual AI checks only run if LiteLLM API key is present, which is mostly CI-only

Visual AI: Do Not Fail, Record

  1. For each found visual issue

    1. spec filename, label, message text

  2. Check Jira using filename + label

  3. Use AI text similarity to find if the found issue was already recorded

    1. If not found, open a Jira issue

  4. Periodically triage the Jira board

Future Work

  • New AI models

  • Automatic screenshot selections

  • Look up page requirements based on specs

Gleb Bahmutov

Thank You

Does This Look Ok To You, Computer?

By Gleb Bahmutov

Does This Look Ok To You, Computer?

There are functional end-to-end tests: a script that drives the browser and uses the app as a real user. But what about visual appearance? Image diffing handles that by locking the web app to a consistent appearance. But what about unexpected issues that might crop up, like an overflowing label? There is now a new technique: asking an AI system to analyze an app screenshot to see if there are potential issues.

  • 21