What Playing Wordle Can Teach Us About Web Testing

Gleb Bahmutov

Sr Director of Engineering

survival is possible* but we need to act now

  • change your life
  • change your bank
  • join an organization
  • vote and protest

3:01am

Sometimes, it takes a few guesses, even with a hint πŸ€ͺ

Sometimes it is the British spelling πŸ‘‘

Sometimes I feel watched

🍺

Spoiler alert 🚨

If I still cannot guess correctly

Want to subscribe to Wordle hint? $1 per month only

Cypress Dashboard with CI results

(show the run or go down)

The recording of the Cypress solving the Wordle

Ran on GitHub Actions, recorded to Cypress Dashboard

Cypress hint screenshot with just one revealed letter

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Agenda

  • Intro to Wordle

  • "Solving" Wordle

  • Solving Wordle

  • The word list

  • What day is today?

  • Send me an email

  • Playing Wordle variants

  • Wordle on CI

Motivation

Playwright: Wordle, with no hands!

There must be a better way

$ npm install -D cypress
it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})
$ npx cypress open

New!  🎁

"Solve" Wordle

"Cypress Wordle" YouTube Playlist has 17 videos

https://www.youtube.com/playlist?list=PLP9o9QNnQuAaihgCPlXyzlj_P-1TTbj-O

describe('Wordle', () => {
  it('loads', () => {
    cy.visit('/index.html')
      .its('localStorage.nyt-wordle-state')
      .then(JSON.parse)
      .its('solution')
      .then((word) => {
        cy.get('game-icon[icon=close]:visible').click()
        word.split('').forEach((letter) => {
          cy.window().trigger('keydown', { key: letter })
        })
        cy.window().trigger('keydown', { key: 'Enter' })
      })
  })
})  

This was a genius solution

πŸ‘Β  Β  Β πŸ‘Ž

It Is All About Trade Offs

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • ShadowDOM ~

  • JS / TS only

The game gives feedback for each letter

Solve Wordle for real

Really Solve Wordle

  1. Start with a list of 5 letter words
  2. Pick a word*
  3. Enter the picked word
  4. Take the game's feedback for each letter
    1. If every letter is correct πŸŽ‰πŸŽ‰πŸŽ‰
    2. Filter the word list**
    3. Go to step 1

* Various strategies

** Edge cases with repeated letters

cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })

Start the test

import { enterWord, countUniqueLetters } from './utils'

function tryNextWord(wordList) {
  const word = Cypress._.sample(wordList)
  enterWord(word)

  let count = 0
  const seen = new Set()
  cy.get(`game-row[letters=${word}]`)
    .find('game-tile')
    .should('have.length', word.length)
    .each(($tile, k) => {
      const letter = $tile.attr('letter')
      if (seen.has(letter)) {
        return
      }
      seen.add(letter)
      const evaluation = $tile.attr('evaluation')

      if (evaluation === 'absent') {
        wordList = wordList.filter((w) => !w.includes(letter))
      } else if (evaluation === 'present') {
        wordList = wordList.filter((w) => w.includes(letter))
      } else if (evaluation === 'correct') {
        count += 1
        wordList = wordList.filter((w) => w[k] === letter)
      }
    })
    .then(() => {
      // after we have entered the word and looked at the feedback
      // we can decide if we solved it, or need to try the next word
      if (count === countUniqueLetters(word)) {
        cy.log('**SOLVED**')
      } else {
        tryNextWord(wordList)
      }
    })
}

The word list gets shorter and shorter with each guess

** Edge cases with repeated letters

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

Q: Where did you get the word list?

A: Where does the application get the word list?

  • Open the DevTools
  • Search the app JavaScript code for "weary"
  • Find the word list

Cypress Network Control

cy.intercept('GET', '/todos').as('load')
cy.visit('/')
cy.wait('@load').its('response.body').then(todos => {
  ...
})

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Spying

https://on.cypress.io/intercept

Cypress Network Control

cy.intercept('GET', '/todos', []).as('load')
cy.visit('/')
cy.wait('@load') // there are zero todos

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Stubbing

Cypress Network Control

cy.intercept('GET', '/todos', { fixture: 'three.json' }).as('load')
cy.visit('/')
cy.wait('@load') // there are 3 todos

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Stubbing

Cypress Network Control

cy.intercept('GET', '/todos', (req) => {
  req.continue(res => {
    res.body.length = 1
  })
}).as('load')
cy.visit('/')
cy.wait('@load') // keep 1 real todo

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Modifying

Cypress Network Control

it('replaces the page title', () => {
  cy.intercept('GET', '**/index.html', (req) => {
    req.continue((res) => {
      res.body = res.body.replace(
        '<title>Wordle - The New York Times</title>',
        '<title>Solve</title>',
      )
    })
  }).as('doc')
  cy.visit('/index.html')
  cy.title().should('equal', 'Solve')
})

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })
// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })
// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control πŸŽ‰

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

"Play Wordle From Any Date Using cy.clock" https://www.youtube.com/watch?v=ZmcOFr2UzZU

Cypress._.range(1, 31).forEach((day) => {
  const date = `2022-01-${day}`

  it(`plays the word from ${date}`, () => {
    cy.clock(Date.UTC(2022, 0, day), ['Date'])
    // play the game
  })
})

Play every Wordle in January 2022

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control πŸŽ‰

  • Clock control πŸ•°

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

Email The Hint

tryNextWord(wordList).then((word) => {
  cy.get('game-tile[letter]', silent).each(($gameTile) => {
    cy.wrap($gameTile, silent)
      .find('.tile', silent)
      .invoke(silent, 'text', '')
  })

  const hint = pickHints(word, numberOfHints)
  // the hint will be something like "__c_a"
  // let's reveal the tiles containing the letters
  hint.split('').forEach((letter, index) => {
    if (letter !== maskLetter) {
      cy.get(`game-row[letters=${word}]`)
        .find('game-tile[letter]')
        .eq(index)
        .find('.tile')
        .invoke('text', letter)
    }
  })
})

Email The Hint

tryNextWord(wordList).then((word) => {
  cy.get('game-tile[letter]', silent).each(($gameTile) => {
    cy.wrap($gameTile, silent)
      .find('.tile', silent)
      .invoke(silent, 'text', '')
  })

  const hint = pickHints(word, numberOfHints)
  // the hint will be something like "__c_a"
  // let's reveal the tiles containing the letters
  hint.split('').forEach((letter, index) => {
    if (letter !== maskLetter) {
      cy.get(`game-row[letters=${word}]`)
        .find('game-tile[letter]')
        .eq(index)
        .find('.tile')
        .invoke('text', letter)
    }
  })
})

Email The Hint

cy.get('#board-container')
  .should('be.visible')
  .screenshot('solved', { overwrite: true })
  .then(() => {
    const screenshot = `${Cypress.spec.name}/solved.png`
    cy.task('sendHintEmail', { screenshot, hint })
  })

Spec running in the browser calls "cy.task" to call Node code to actually send the email with the screenshot

Email The Hint

const sgMail = require('@sendgrid/mail')
sgMail.setApiKey(process.env.SENDGRID_API_KEY)
async sendHintEmail({ screenshot, hint }) {
  const screenshotPath = path.join(config.screenshotsFolder, screenshot)
  const imageBase64 = fs.readFileSync(screenshotPath).toString('base64')
  const msg = {
    to: process.env.WORDLE_HINT_EMAIL,
    from: process.env.SENDGRID_FROM,
    subject: 'Wordle daily hint',
    text: `Today's hint: ${hint}`,
    html: `
      <div>Today's hint: <pre>${hint}</pre></div>
      <div><img src="data:image/png;base64,${imageBase64}" /></div>
      <div>Solved by <a href="https://github.com/bahmutov/cypress-wordle">cypress-wordle</a></div>
    `,
  }
  const response = await sgMail.send(msg)
}

regular Node code to send an email (plugin file or cypress.config.js)

Alternative: almost solved

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control πŸŽ‰

  • Clock control πŸ•°

  • Node.js in plugins file

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

  • Spec vs plugins file

Wordle clones and variants

  • Each variant has its own page structure

Abstract page access

Reuse the solution algorithm

// solves the Vue version of the Wordle game
// https://github.com/yyx990803/vue-wordle
import { Playing } from './pages'
import { solve } from '../utils/solver'
describe('Vue Wordle', { baseUrl: 'https://vue-wordle.netlify.app/' }, () => {
  beforeEach(() => {
    cy.fixture('wordlist.json').as('wordList')
  })

  it('finds the target word starting with another word', () => {
    // the word to guess
    const word = 'quick'
    cy.visit(`/?${btoa(word)}`)
    solve('start', Playing).should('equal', word)
  })
})
export function solve(startWord, pageObject) {
  expect(pageObject, 'page object')
    .to.be.an('object')
    .and.to.respondTo('enterWord')
    .and.to.respondTo('getLetters')

  return cy
    .get('@wordList')
    .then((wordList) => 
      tryNextWord(wordList, startWord, pageObject))
}

The solution algorithm (1/2)

function tryNextWord(wordList, word, pageObject) {
  if (!word) {
    word = pickWordWithUniqueLetters(wordList)
  }
  pageObject.enterWord(word)

  return pageObject.getLetters(word).then((letters) => {
    wordList = updateWordList(wordList, word, letters)
    if (wordList === word) {
      // we solved it!
      return word
    }
    return tryNextWord(wordList, null, pageObject)
  })
}

The solution algorithm (2/2)

export const Playing = {
  enterWord(word) {
    word.split('').forEach((letter) => {
      cy.window(silent).trigger('keyup', { key: letter })
    })
    cy.window(silent)
      .trigger('keyup', { key: 'Enter' })
  },
  getLetters(word) {
    return cy
      .get('#board .row .tile.filled.revealed .back')
      .should('have.length.gte', word.length)
      .then(($tiles) => {
        // only take the last 5 letters
        return $tiles
          .toArray()
          .slice(-5)
          .map((tile, k) => {
            const letter = tile.innerText.toLowerCase()
            const evaluation = tile.classList.contains('correct')
              ? 'correct'
              : tile.classList.contains('present')
              ? 'present'
              : 'absent'
            console.log('%d: letter %s is %s', k, letter, evaluation)
            return { k, letter, evaluation }
          })
      })
  }
}  

The page object (for Vue Wordle)

enterword(word) {
  ...
},
getLetters(word) {
  return cy
    .get(`game-row[letters=${word}]`)
    .find('game-tile', silent)
    .should('have.length', word.length)
    .then(($tiles) => {
      return $tiles.toArray().map((tile, k) => {
        const letter = tile.getAttribute('letter')
        const evaluation = tile.getAttribute('evaluation')
        return { k, letter, evaluation }
      })
    })
},

NYTimes / original Wordle Page Object

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control πŸŽ‰

  • Clock control πŸ•°

  • Node.js in plugins file

  • Page objects, functions, custom commands

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

  • Spec vs plugins file

Play Worlde on CI

Play Worlde on CI

name: hint
# run the workflow every morning
# or when we trigger the workflow manually
on:
  workflow_dispatch:
    inputs:
      email:
        description: Email to send the hint to
        type: string
        default: ''
        required: false
      hints:
        description: Number of letters to reveal
        type: integer
        default: 1
  schedule:
    # UTC time
    - cron: '0 7 * * *'
jobs:
  hint:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout πŸ›Ž
        uses: actions/checkout@v3

      - name: Cypress run πŸ§ͺ
        #  https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v3
        with:
          spec: 'cypress/integration/email-hint.js'
          env: 'hints=${{ github.event.inputs.hints }}'
        env:
          SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
          SENDGRID_FROM: ${{ secrets.SENDGRID_FROM }}
          WORDLE_HINT_EMAIL: ${{ github.event.inputs.email || secrets.WORDLE_HINT_EMAIL }}

Run all Wordle tests on CI

Run all Wordle tests on CI

name: ci
on: [push]
jobs:
  test:
    uses: bahmutov/cypress-workflows/.github/workflows/parallel.yml@v1
    with:
      n: 5 # use 5 machines in parallel
    secrets:
      recordKey: ${{ secrets.CYPRESS_RECORD_KEY }}

Cypress Dashboard run load balances all spec files

https://dashboard.cypress.io/projects/6iu6px/

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control πŸŽ‰

  • Clock control πŸ•°

  • Node.js in plugins file

  • Page objects, functions, custom commands

  • CI integration(s) βœ…

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm πŸŒ€

  • Spec vs plugins file

Playing Wordle

is FUN

Know what to expect

Run it on CI

Early and all the time

Games are an excellent testing demo project for your job search

Thank You πŸ‘

Gleb Bahmutov

Sr Director of Engineering

What Playing Wordle Can Teach Us About Web Testing

By Gleb Bahmutov

What Playing Wordle Can Teach Us About Web Testing

A new online game Wordle has become very popular recently. In the game, you enter five letter words until you match the secret word. The game responds by giving you clues about each letter you have entered. How would you test this online game using an end-to-end test runner like Cypress? Could you solve it using a test? In this presentation, I will show how we can test the game using Cypress most powerful features. Presented at DevReach Boston Sept 2022, watch at https://www.youtube.com/watch?v=CwHciuR5udo

  • 1,820