What Playing Wordle Can Teach Us About Web Testing

https://slides.com/bahmutov/wordle-testing

Gleb Bahmutov

Sr Director of Engineering

📺 https://www.youtube.com/watch?v=CwHciuR5udo

survival is possible* but we need to act now

  • change your life
  • change your bank
  • join an organization
  • vote and protest

@bahmutov

rebellion.global www.wedonthavetime.org 

www.sierraclub.org 350.org www.greenpeace.org 

www.sunrisemovement.org

3:01am

@bahmutov

Sometimes, it takes a few guesses, even with a hint 🤪

Sometimes it is the British spelling 👑

Sometimes I feel watched

🍺

Spoiler alert 🚨

If I still cannot guess correctly

Want to subscribe to Wordle hint? $1 per month only

https://github.com/bahmutov/cypress-wordle

@bahmutov

Cypress Dashboard with CI results

https://dashboard.cypress.io/projects/6iu6px/runs

(show the run or go down)

The recording of the Cypress solving the Wordle

Ran on GitHub Actions, recorded to Cypress Dashboard

Cypress hint screenshot with just one revealed letter

@bahmutov

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

Agenda

  • Intro to Wordle

  • "Solving" Wordle

  • Solving Wordle

  • The word list

  • What day is today?

  • Send me an email

  • Playing Wordle variants

  • Wordle on CI

Motivation

https://www.tomcools.be/post/jan-2022-playwright-wordle/

Playwright: Wordle, with no hands!

(show the site https://github.com/TomCools/playwright-wordle)

There must be a better way

$ npm install -D cypress

@bahmutov

it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

https://on.cypress.io/api

@bahmutov

https://github.com/cypress-io/cypress-example-todomvc

$ npx cypress open

@bahmutov

https://github.com/cypress-io/cypress-example-todomvc

https://learn.cypress.io/

New!  🎁

"Solve" Wordle

@bahmutov

https://www.youtube.com/watch?v=pzFzOKEV-eo 6 minutes

"Cypress Wordle" YouTube Playlist has 17 videos

https://www.youtube.com/playlist?list=PLP9o9QNnQuAaihgCPlXyzlj_P-1TTbj-O

describe('Wordle', () => {
  it('loads', () => {
    cy.visit('/index.html')
      .its('localStorage.nyt-wordle-state')
      .then(JSON.parse)
      .its('solution')
      .then((word) => {
        cy.get('game-icon[icon=close]:visible').click()
        word.split('').forEach((letter) => {
          cy.window().trigger('keydown', { key: letter })
        })
        cy.window().trigger('keydown', { key: 'Enter' })
      })
  })
})

https://github.com/bahmutov/cypress-wordle/blob/main/cypress/integration/spec.js

@bahmutov

This was a genius solution

👍     👎

It Is All About Trade Offs

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • ShadowDOM ~

  • JS / TS only

@bahmutov

The game gives feedback for each letter

Solve Wordle for real

Really Solve Wordle

  1. Start with a list of 5 letter words
  2. Pick a word*
  3. Enter the picked word
  4. Take the game's feedback for each letter
    1. If every letter is correct 🎉🎉🎉
    2. Filter the word list**
    3. Go to step 1

* Various strategies

** Edge cases with repeated letters

@bahmutov

https://github.com/bahmutov/cypress-wordle/blob/main/cypress/integration/solve.js​

@bahmutov

cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })

Start the test

https://github.com/bahmutov/cypress-wordle/blob/main/cypress/integration/solve.js​

@bahmutov

import { enterWord, countUniqueLetters } from './utils'

function tryNextWord(wordList) {
  const word = Cypress._.sample(wordList)
  enterWord(word)

  let count = 0
  const seen = new Set()
  cy.get(`game-row[letters=${word}]`)
    .find('game-tile')
    .should('have.length', word.length)
    .each(($tile, k) => {
      const letter = $tile.attr('letter')
      if (seen.has(letter)) {
        return
      }
      seen.add(letter)
      const evaluation = $tile.attr('evaluation')

      if (evaluation === 'absent') {
        wordList = wordList.filter((w) => !w.includes(letter))
      } else if (evaluation === 'present') {
        wordList = wordList.filter((w) => w.includes(letter))
      } else if (evaluation === 'correct') {
        count += 1
        wordList = wordList.filter((w) => w[k] === letter)
      }
    })
    .then(() => {
      // after we have entered the word and looked at the feedback
      // we can decide if we solved it, or need to try the next word
      if (count === countUniqueLetters(word)) {
        cy.log('**SOLVED**')
      } else {
        tryNextWord(wordList)
      }
    })
}

The word list gets shorter and shorter with each guess

** Edge cases with repeated letters

@bahmutov

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

@bahmutov

Q: Where did you get the word list?

A: Where does the application get the word list?

  • Open the DevTools
  • Search the app JavaScript code for "weary"
  • Find the word list

Cypress Network Control

cy.intercept('GET', '/todos').as('load')
cy.visit('/')
cy.wait('@load').its('response.body').then(todos => {
  ...
})

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Spying

https://on.cypress.io/intercept

Cypress Network Control

cy.intercept('GET', '/todos', []).as('load')
cy.visit('/')
cy.wait('@load') // there are zero todos

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Stubbing

Cypress Network Control

cy.intercept('GET', '/todos', { fixture: 'three.json' }).as('load')
cy.visit('/')
cy.wait('@load') // there are 3 todos

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Stubbing

Cypress Network Control

cy.intercept('GET', '/todos', (req) => {
  req.continue(res => {
    res.body.length = 1
  })
}).as('load')
cy.visit('/')
cy.wait('@load') // keep 1 real todo

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

Modifying

Cypress Network Control

it('replaces the page title', () => {
  cy.intercept('GET', '**/index.html', (req) => {
    req.continue((res) => {
      res.body = res.body.replace(
        '<title>Wordle - The New York Times</title>',
        '<title>Solve</title>',
      )
    })
  }).as('doc')
  cy.visit('/index.html')
  cy.title().should('equal', 'Solve')
})

https://on.cypress.io/intercept

Cypress Network Testing Exercises https://cypress.tips/courses/network-testing

Cypress Network Control

// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })
// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })

@bahmutov

// look up the word list in the JavaScript bundle
// served by the application
cy.intercept('GET', '**/main.*.js', (req) => {
  req.continue((res) => {
    // by inserting a variable assignment here
    // we will get the reference to the list on the window object
    // which is reachable from this test
    res.body = res.body.replace('=["cigar', '=window.wordList=["cigar')
  })
})
cy.visit('/index.html')
  // the "window.wordList" variable is now available
  // that will be our initial list of words
  .its('wordList')
  .then((wordList) => {
    cy.get('game-icon[icon=close]:visible').click()
    tryNextWord(wordList)
  })

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control 🎉

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

@bahmutov

"Play Wordle From Any Date Using cy.clock" https://www.youtube.com/watch?v=ZmcOFr2UzZU

@bahmutov

Cypress._.range(1, 31).forEach((day) => {
  const date = `2022-01-${day}`

  it(`plays the word from ${date}`, () => {
    cy.clock(Date.UTC(2022, 0, day), ['Date'])
    // play the game
  })
})

https://github.com/bahmutov/cypress-wordle/blob/main/cypress/integration/play-every-day.js

Play every Wordle in January 2022

https://www.youtube.com/watch?v=5X4RuyEoQgY

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control 🎉

  • Clock control 🕰

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

@bahmutov

Email The Hint

tryNextWord(wordList).then((word) => {
  cy.get('game-tile[letter]', silent).each(($gameTile) => {
    cy.wrap($gameTile, silent)
      .find('.tile', silent)
      .invoke(silent, 'text', '')
  })

  const hint = pickHints(word, numberOfHints)
  // the hint will be something like "__c_a"
  // let's reveal the tiles containing the letters
  hint.split('').forEach((letter, index) => {
    if (letter !== maskLetter) {
      cy.get(`game-row[letters=${word}]`)
        .find('game-tile[letter]')
        .eq(index)
        .find('.tile')
        .invoke('text', letter)
    }
  })
})

Email The Hint

tryNextWord(wordList).then((word) => {
  cy.get('game-tile[letter]', silent).each(($gameTile) => {
    cy.wrap($gameTile, silent)
      .find('.tile', silent)
      .invoke(silent, 'text', '')
  })

  const hint = pickHints(word, numberOfHints)
  // the hint will be something like "__c_a"
  // let's reveal the tiles containing the letters
  hint.split('').forEach((letter, index) => {
    if (letter !== maskLetter) {
      cy.get(`game-row[letters=${word}]`)
        .find('game-tile[letter]')
        .eq(index)
        .find('.tile')
        .invoke('text', letter)
    }
  })
})

Email The Hint

cy.get('#board-container')
  .should('be.visible')
  .screenshot('solved', { overwrite: true })
  .then(() => {
    const screenshot = `${Cypress.spec.name}/solved.png`
    cy.task('sendHintEmail', { screenshot, hint })
  })

@bahmutov

Spec running in the browser calls "cy.task" to call Node code to actually send the email with the screenshot

Email The Hint

const sgMail = require('@sendgrid/mail')
sgMail.setApiKey(process.env.SENDGRID_API_KEY)
async sendHintEmail({ screenshot, hint }) {
  const screenshotPath = path.join(config.screenshotsFolder, screenshot)
  const imageBase64 = fs.readFileSync(screenshotPath).toString('base64')
  const msg = {
    to: process.env.WORDLE_HINT_EMAIL,
    from: process.env.SENDGRID_FROM,
    subject: 'Wordle daily hint',
    text: `Today's hint: ${hint}`,
    html: `
      <div>Today's hint: <pre>${hint}</pre></div>
      <div><img src="data:image/png;base64,${imageBase64}" /></div>
      <div>Solved by <a href="https://github.com/bahmutov/cypress-wordle">cypress-wordle</a></div>
    `,
  }
  const response = await sgMail.send(msg)
}

regular Node code to send an email (plugin file or cypress.config.js)

https://github.com/bahmutov/cypress-wordle/actions

Alternative: almost solved

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control 🎉

  • Clock control 🕰

  • Node.js in plugins file

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

  • Spec vs plugins file

@bahmutov

Wordle clones and variants

  • Each variant has its own page structure

Abstract page access

Reuse the solution algorithm

// solves the Vue version of the Wordle game
// https://github.com/yyx990803/vue-wordle
import { Playing } from './pages'
import { solve } from '../utils/solver'
describe('Vue Wordle', { baseUrl: 'https://vue-wordle.netlify.app/' }, () => {
  beforeEach(() => {
    cy.fixture('wordlist.json').as('wordList')
  })

  it('finds the target word starting with another word', () => {
    // the word to guess
    const word = 'quick'
    cy.visit(`/?${btoa(word)}`)
    solve('start', Playing).should('equal', word)
  })
})
export function solve(startWord, pageObject) {
  expect(pageObject, 'page object')
    .to.be.an('object')
    .and.to.respondTo('enterWord')
    .and.to.respondTo('getLetters')

  return cy
    .get('@wordList')
    .then((wordList) => 
      tryNextWord(wordList, startWord, pageObject))
}

The solution algorithm (1/2)

function tryNextWord(wordList, word, pageObject) {
  if (!word) {
    word = pickWordWithUniqueLetters(wordList)
  }
  pageObject.enterWord(word)

  return pageObject.getLetters(word).then((letters) => {
    wordList = updateWordList(wordList, word, letters)
    if (wordList === word) {
      // we solved it!
      return word
    }
    return tryNextWord(wordList, null, pageObject)
  })
}

The solution algorithm (2/2)

export const Playing = {
  enterWord(word) {
    word.split('').forEach((letter) => {
      cy.window(silent).trigger('keyup', { key: letter })
    })
    cy.window(silent)
      .trigger('keyup', { key: 'Enter' })
  },
  getLetters(word) {
    return cy
      .get('#board .row .tile.filled.revealed .back')
      .should('have.length.gte', word.length)
      .then(($tiles) => {
        // only take the last 5 letters
        return $tiles
          .toArray()
          .slice(-5)
          .map((tile, k) => {
            const letter = tile.innerText.toLowerCase()
            const evaluation = tile.classList.contains('correct')
              ? 'correct'
              : tile.classList.contains('present')
              ? 'present'
              : 'absent'
            console.log('%d: letter %s is %s', k, letter, evaluation)
            return { k, letter, evaluation }
          })
      })
  }
}

The page object (for Vue Wordle)

enterword(word) {
  ...
},
getLetters(word) {
  return cy
    .get(`game-row[letters=${word}]`)
    .find('game-tile', silent)
    .should('have.length', word.length)
    .then(($tiles) => {
      return $tiles.toArray().map((tile, k) => {
        const letter = tile.getAttribute('letter')
        const evaluation = tile.getAttribute('evaluation')
        return { k, letter, evaluation }
      })
    })
},

NYTimes / original Wordle Page Object

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control 🎉

  • Clock control 🕰

  • Node.js in plugins file

  • Page objects, functions, custom commands

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

  • Spec vs plugins file

@bahmutov

Play Worlde on CI

@bahmutov

Play Worlde on CI

name: hint
# run the workflow every morning
# or when we trigger the workflow manually
on:
  workflow_dispatch:
    inputs:
      email:
        description: Email to send the hint to
        type: string
        default: ''
        required: false
      hints:
        description: Number of letters to reveal
        type: integer
        default: 1
  schedule:
    # UTC time
    - cron: '0 7 * * *'
jobs:
  hint:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v3

      - name: Cypress run 🧪
        #  https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v3
        with:
          spec: 'cypress/integration/email-hint.js'
          env: 'hints=${{ github.event.inputs.hints }}'
        env:
          SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
          SENDGRID_FROM: ${{ secrets.SENDGRID_FROM }}
          WORDLE_HINT_EMAIL: ${{ github.event.inputs.email || secrets.WORDLE_HINT_EMAIL }}

Run all Wordle tests on CI

Run all Wordle tests on CI

name: ci
on: [push]
jobs:
  test:
    uses: bahmutov/cypress-workflows/.github/workflows/parallel.yml@v1
    with:
      n: 5 # use 5 machines in parallel
    secrets:
      recordKey: ${{ secrets.CYPRESS_RECORD_KEY }}

https://github.com/bahmutov/cypress-workflows

@bahmutov

Cypress Dashboard run load balances all spec files

https://dashboard.cypress.io/projects/6iu6px/

@bahmutov

Cypress

Pros

Cons

  • Fast install

  • Access app data

  • Time-travel

  • Screenshots, videos

  • JS / TS only

  • Network control 🎉

  • Clock control 🕰

  • Node.js in plugins file

  • Page objects, functions, custom commands

  • CI integration(s) ✅

  • ShadowDOM ~

  • JS / TS only

  • Recursive algorithm 🌀

  • Spec vs plugins file

@bahmutov

Playing Wordle

is FUN

Know what to expect

Run it on CI

Early and all the time

Games are an excellent testing demo project for your job search

Thank You 👏

https://slides.com/bahmutov/wordle-testing

Code https://github.com/bahmutov/cypress-wordle

Gleb Bahmutov

Sr Director of Engineering

Learn more: https://cypress.tips/ and https://docs.cypress.io/

https://cypress.tips/

What Playing Wordle Can Teach Us About Web Testing

By Gleb Bahmutov

What Playing Wordle Can Teach Us About Web Testing

A new online game Wordle has become very popular recently. In the game, you enter five letter words until you match the secret word. The game responds by giving you clues about each letter you have entered. How would you test this online game using an end-to-end test runner like Cypress? Could you solve it using a test? In this presentation, I will show how we can test the game using Cypress most powerful features. Presented at DevReach Boston Sept 2022, watch at https://www.youtube.com/watch?v=CwHciuR5udo

  • 1,835

More from Gleb Bahmutov