Testing Your Documentation Search

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

@bahmutov

survival is possible* but we need to act now

change your life
join an organization

@bahmutov

rebellion.global 350.org

Talk Contents

The money funnel
Good documentation
Documentation search with Algolia
Testing the documentation search
- "blind", using fixture, spying on network
Tips & Tricks

The Funnel

aka "You are always losing customers"

@bahmutov

The Funnel

aka "You are always losing customers"

💵

💰

👀

🧑‍💻

🦸

@bahmutov

The Funnel

aka "You are always losing customers"

10-20%

@bahmutov

The Funnel

aka "You are always losing customers"

10-20%

@bahmutov

The Funnel

aka "You are always losing customers"

10-20%

@bahmutov

The Funnel

aka "You are always losing customers"

This tool does not do what I need

It does not work on my platform

Valid reasons

@bahmutov

The Funnel

aka "You are always losing customers"

This tool does not do what I need

It does not work on my platform

Valid reasons

This tool does not do what I need (but it does)

Does not work on my platform (but it does work)

Wrong reasons

@bahmutov

The Funnel

aka "You are always losing customers"

This tool does not run on my CI

Valid reasons

I don't know how to make this tool work on my CI

Wrong reasons

@bahmutov

The Funnel

aka "You are always losing customers"

I cannot afford this tool

Valid reasons

I don't know what value this tool gives me

Wrong reasons

The Funnel

aka "You are always losing customers"

I am happy with this tool, that's enough for me

Valid reasons

I did not know this tool could do that

Wrong reasons

The Funnel

aka "You are always losing customers"

Market fit

Features

Bug fixes

@bahmutov

The Funnel

aka "You are always losing customers"

Market fit

Features

Bug fixes

Bad UI

Bad docs

Lack of examples

@bahmutov

The Funnel

aka "You are always losing customers"

Market fit

Features

Bug fixes

Bad UI

Bad docs

Lack of examples

Goal: drive the "bad" area to zero

@bahmutov

The Funnel

aka "You are always losing customers"

Bad UI

Bad docs

Lack of examples

@bahmutov

Goal: drive the "bad" area to zero

Features vs Docs

Bad UI

Bad docs

Lack of examples

⚖️

Market fit

Features

Bug fixes

So
Much
Easier

@bahmutov

Documentation makes or breaks projects

Improve the Docs: Personas

@bahmutov

First time visitor

First time user

Repeat user

Paying customer

Promoter

🔑 Visitor / User should always find the right answer

First time visitor

What does this tool / service do?
Will it work for me?

@bahmutov

🔑 Visitor / User should always find the right answer

First time user

How do I do X?
How do I do Y?
What do I do for error A?

First time user

🔑 Visitor / User should always find the right answer

Repeat user

How do I do X and Y and Z?
What do I do for error A, B, C?
How do I run using ...?
How do I learn more?

Repeat user

🔑 Visitor / User should always find the right answer

Paying customer

same as repeat user +
Is there training?
I got stuck, is there help?

Paying customer

🔑 Visitor / User should always find the right answer

Promoter

When are the new releases?
What else can this tool do?
Can I suggest a feature?
Is there an email list or webinars?

Docs keep growing

Information for every user persona

@bahmutov

How to measure how good your documentation is ... and start crying

https://gitter.im/cypress-io/cypress

Is there a way to disable command snapshots?

How to preserve session cookie?

How to check the length of text?

How to measure how good your documentation is ... and start crying

https://gitter.im/cypress-io/cypress

Every question here is a failure of the documentation

Every question here is probably a:

a support issue 💵
maybe a lost customer 💸

Documentation miss percentage

https://gitter.im/cypress-io/cypress

all well-defined questions across chat, support emails, github issues

questions that have answer in your documentation

* 100%

Documentation miss percentage

< 50% write more docs!

≥ 50%

docs have a problem

(many things are not documented)

(users cannot find the right documentation)

You Want Users to Find Answers to Their Questions by Themselves

@bahmutov

The documentation demands for different user personas are contradictory

Why Good Docs Are Hard?

@bahmutov

First time visitor

First time user

Repeat user

Paying customer

Promoter

💥

"show me Hello, World!"

"show me the changelog diff from version X to Y"

"show me a tutorial"

"show me how to do X"

"how do I solve my issue or bug?"

👀 Optimize documentation structure for beginners

"Hello World" example front and center
There are 10 beginners for every 1 advanced user. Every user starts as a beginner

@bahmutov

👀 Optimize documentation structure for beginners

Then add all your documentation to the site reachable from the index

@bahmutov

💡 Make a powerful docs search for every persona

https://docs.cypress.io/

Sample App

https://glebbahmutov.com/blog/scrape-static-site-with-algolia/

https://glebbahmutov.com/triple-tested/

Why not use the built-in client-side indexing?

Hint: look at the search results for "about" query

Algolia

https://docsearch.algolia.com/

I am unaffiliated, just a user
Generous free plan
Good documentation, scraping tools, nice UI

@bahmutov

Algolia in action

@bahmutov

Create Algolia App

https://www.algolia.com/account/applications

@bahmutov

Create Search Index

https://www.algolia.com/account/applications

@bahmutov

Workflow

After every deploy 🔼
1. Scrape the site 🕷
Set search widget to point at the index using Algolia SDKs

@bahmutov

API Keys

to add records to the index

@bahmutov

Keep Admin API key private!

API Keys

to search index from site

(public)

@bahmutov

Scraping the Site

@bahmutov

Document Structure

{
  "index_name": "scrape-test",
  "start_urls": ["https://glebbahmutov.com/triple-tested/"],
  "selectors": {
    "lvl0": {
      "selector": ".site-name",
      "global": true
    },
    "lvl1": ".content__default h1",
    "lvl2": ".content__default h2",
    "lvl3": ".content__default h3",
    "lvl4": ".content__default h4",
    "lvl5": ".content__default h5",
    "text": ".content__default p, .content__default li"
  }
}

Algolia config (JSON)

Scraping

# when scraping the site, inject secrets as environment variables
# then pass their values into the Docker container using "-e" syntax
# and inject config.json contents as another variable
- name: scrape the site 🧽
  env:
    APPLICATION_ID: ${{ secrets.APPLICATION_ID }}
    API_KEY: ${{ secrets.API_KEY }}
  run: |
    docker run \
    -e APPLICATION_ID -e API_KEY \
    -e CONFIG="$(cat config.json)" \
    algolia/docsearch-scraper:v1.6.0

use Algolia Docker image

https://github.com/bahmutov/scrape-test

@bahmutov

Scraping

# when scraping the site, inject secrets as environment variables
# then pass their values into the Docker container using "-e" syntax
# and inject config.json contents as another variable
- name: scrape the site 🧽
  env:
    APPLICATION_ID: ${{ secrets.APPLICATION_ID }}
    API_KEY: ${{ secrets.API_KEY }}
  run: |
    docker run \
    -e APPLICATION_ID -e API_KEY \
    -e CONFIG="$(cat config.json)" \
    algolia/docsearch-scraper:v1.6.0

use Algolia Docker image

https://github.com/bahmutov/scrape-test

@bahmutov

Larger sites: Multiple roots

blog

examples

Larger sites: Multiple roots

{
  "index_name": "cypress",
  "start_urls": [
    {
      "url": "https://docs.cypress.io/",
      "page_rank": 10
    },
    {
      "url": "https://example.cypress.io/",
      "selectors_key": "kitchensink",
      "tags": ["example"],
      "page_rank": 2
    },
    {
      "url": "https://www.cypress.io/blog/",
      "selectors_key": "blog",
      "tags": ["blog post"],
      "page_rank": 1
    }
  ],
  "stop_urls": [
    "^https://docs.cypress.io/ja/",
    "^https://docs.cypress.io/zh-cn/",
    "^https://docs.cypress.io/pt-br/",
    "^https://docs.cypress.io/ru/"
  ],
  "selectors_exclude": [],
  "selectors": {
    "default": {
      "lvl0": "article h1.article-title",
      "lvl1": "article h1.article-heading",
      "lvl2": "article h2.article-heading",
      "text": "article .article-content p, article .article-content tr, article .article-content li, article .article-content pre"
    },
    "blog": {
      "lvl0": "article h1",
      "lvl1": "article h2",
      "text": "article p, article pre"
    },
    "kitchensink": {
      "lvl0": ".container h1",
      "lvl1": ".container h4",
      "text": ".container p, .container pre"
    }
  },
  "nb_hits": 0,
  "min_indexed_level": 2
}

Algolia config (JSON)

Larger sites: Multiple roots

2 minutes to scrape, creates ~ 21k records

@bahmutov

<ais-search-box
  // Optional parameters
  placeholder="string"
  submitTitle="string"
  resetTitle="string"
  [searchAsYouType]="boolean"
  [autofocus]="boolean"
></ais-search-box>

ais-search-box

https://www.algolia.com/doc/api-reference/widgets/search-box/angular

UI

Deploy 🏗
Scrape 🕷
Test 🔎

Let's test the search

describe('Angular Doc Search', () => {
  it('shows native results', () => {
    cy.visit('/', {
      onBeforeLoad(win) {
        // ServiceWorker messes up with the page load
        delete win.navigator.__proto__.serviceWorker
      }
    })
    // delay each keystroke for the demo
    cy.get('input[aria-label=search]').type('testing', {delay: 70})

    // six search results columns
    cy.get('.search-section-header').should('have.length', 6)
    cy.contains('.search-section-header', 'cli')
      .parent('.search-area')
      .find('.search-page').should('have.length.gte', 3)
  })
})

Let's test the search

a "blind" test guesses the search term and results

describe('Angular Doc Search', () => {
  it('shows native results', () => {
    cy.visit('/', {
      onBeforeLoad(win) {
        // ServiceWorker messes up with the page load
        delete win.navigator.__proto__.serviceWorker
      }
    })
    // delay each keystroke for the demo
    cy.get('input[aria-label=search]').type('testing', {delay: 70})

    // six search results columns
    cy.get('.search-section-header').should('have.length', 6)
    cy.contains('.search-section-header', 'cli')
      .parent('.search-area')
      .find('.search-page').should('have.length.gte', 3)
  })
})

Let's test the search

describe('Angular Doc Search', () => {
  it('shows native results', () => {
    cy.visit('/', {
      onBeforeLoad(win) {
        // ServiceWorker messes up with the page load
        delete win.navigator.__proto__.serviceWorker
      }
    })
    // delay each keystroke for the demo
    cy.get('input[aria-label=search]').type('testing', {delay: 70})

    // six search results columns
    cy.get('.search-section-header').should('have.length', 6)
    cy.contains('.search-section-header', 'cli')
      .parent('.search-area')
      .find('.search-page').should('have.length.gte', 3)
  })
})

Let's test the search

describe('Angular Doc Search', () => {
  it('shows native results', () => {
    cy.visit('/', {
      onBeforeLoad(win) {
        // ServiceWorker messes up with the page load
        delete win.navigator.__proto__.serviceWorker
      }
    })
    // delay each keystroke for the demo
    cy.get('input[aria-label=search]').type('testing', {delay: 70})

    // six search results columns
    cy.get('.search-section-header').should('have.length', 6)
    cy.contains('.search-section-header', 'cli')
      .parent('.search-area')
      .find('.search-page').should('have.length.gte', 3)
  })
})

Let's test the search

Let's Dig Deeper

it('shows single search result', () => {
  // https://on.cypress.io/intercept
  cy.intercept('/search-data.json', { fixture: 'single-result.json' })
  cy.visit('/', {
    onBeforeLoad(win) {
      // ServiceWorker messes up with the page load
      delete win.navigator.__proto__.serviceWorker
    }
  })
  // delay each keystroke for the demo
  cy.get('input[aria-label=search]').type('testing', {delay: 70})
})

[{
  "headingWords": "testing",
  "keywords": "testing unit component e2e",
  "path": "cli/test",
  "title": "Testing is fun",
  "titleWords": "testing is fun",
  "type": "content"
}]

cypress/fixtures/single-result.json

cy.get('input[aria-label=search]')
  .type('testing', { delay: 70 })

cy.contains('.search-section-header', 'cli')
  .parent('.search-area')
  .contains('.search-page', 'Testing is fun')
  .click()

cy.get('input[aria-label=search]')
  .type('testing', { delay: 70 })

cy.contains('.search-section-header', 'cli')
  .parent('.search-area')
  .contains('.search-page', 'Testing is fun')
  .click()
  
 cy.location('pathname').should('equal', '/cli/test')

name: ci
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2

Continuous Integration Break

https://github.com/cypress-io/github-action

🎉 Break is Over 🎊

https://github.com/bahmutov/testing-angular-docs-search/actions

cy.get('input[aria-label=search]')
  .type('testing', { delay: 70 })

cy.contains('.search-section-header', 'cli')
  .parent('.search-area')
  .contains('.search-page', 'Testing is fun')
  .click()
  
cy.location('pathname')
  .should('equal', '/cli/test')

Hardcoded test data

The same data is in the fixture JSON file ...

import singleResult from '../fixtures/single-result.json'

cy.intercept('/search-data.json', singleResult)
cy.visit()

const {headingWords, title, path} = singleResult[0]
cy.get('input[aria-label=search]')
  .type(headingWords, { delay: 70 })

cy.get('.search-section-header')
  .parent('.search-area')
  .contains('.search-page', title)
  .click()

cy.location('pathname')
  .should('equal', '/' + path)

Load test data from a fixture file & intercept too

If test data comes from a fixture ...

then we can use the real data from the network call

cy.intercept('/search-data.json').as('search')
cy.visit()

spy on network call

cy.intercept('/search-data.json').as('search')
cy.visit()
cy.wait('@search').its('response.body')

spy on network call

cy.wait('@search').its('response.body')
  .then(list => {
    return Cypress._.find(list, { title: 'Accessibility in Angular' })
  })
  .then(result => {
    expect(result).to.be.an('object')
  })

.then(result => {
  expect(result).to.be.an('object')
  const { headingWords, title, path } = result
  const search = headingWords.split(' ')[0]
  // delay each keystroke for the demo
  cy.get('input[aria-label=search]')
    .type(search, { delay: 70 })

  cy.contains('.search-page a', title).click()
  cy.location('pathname')
    .should('equal', '/' + path)
})

Talk Contents

The money funnel
Good documentation
Documentation search with Algolia
Testing the documentation search
- "blind", using fixture, spying on network
Tips & Tricks

code comments are not indexed 😟

💡 Include code comments in the index

https://github.com/bahmutov/cypress-examples

@bahmutov

💡 Include code comments in the index

{
  "index_name": "cypress-examples",
  "start_urls": ["https://glebbahmutov.com/cypress-examples/"],
  "selectors": {
    "lvl0": {
      "selector": ".site-name",
      "global": true
    },
    "lvl1": ".content__default h1",
    "lvl2": ".content__default h2",
    "lvl3": ".content__default h3",
    "lvl4": ".content__default h4",
    "lvl5": ".content__default h5",
    "text": ".content__default p, .content__default li, .content__default pre .comment"
  }
}

https://github.com/bahmutov/cypress-examples

algolia-config.json

@bahmutov

scrape paragraphs AND list items AND code comments

💡 Include code comments in the index

{
  "index_name": "cypress-examples",
  "start_urls": ["https://glebbahmutov.com/cypress-examples/"],
  "selectors": {
    "lvl0": {
      "selector": ".site-name",
      "global": true
    },
    "lvl1": ".content__default h1",
    "lvl2": ".content__default h2",
    "lvl3": ".content__default h3",
    "lvl4": ".content__default h4",
    "lvl5": ".content__default h5",
    "text": ".content__default p, .content__default li, .content__default pre .comment"
  }
}

https://github.com/bahmutov/cypress-examples

algolia-config.json

Tip: use $$(selector) in DevTools

💡 Look at searches with no results

⚠️ free Algolia plans only get weekly emails with such queries

write docs for these searches!

@bahmutov

💡 Search from CLI

https://github.com/bahmutov/cy-search

@bahmutov

Final Thoughts

Good documentation is essential to your project's success

@bahmutov

Final Thoughts

Creating an ideal documentation structure is hard and might be impossible

@bahmutov

Final Thoughts

Creating a good enough text search using Algolia is pretty simple 🎉

@bahmutov

Do not skip testing the search 🔎

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

👏 Thank you 👏

Browse slides https://slides.com/bahmutov/testing-docs

Look at code https://github.com/bahmutov/testing-angular-docs-search

Read https://glebbahmutov.com/blog/scrape-static-site-with-algolia/

Follow @bahmutov

Act: on climate

Testing Your Documentation Search

Gleb Bahmutov

VP of Engineering

Cypress.io

@bahmutov

our planet is in imminent danger

survival is possible* but we need to act now

Talk Contents

The money funnel

Good documentation

Documentation search with Algolia

Testing the documentation search

"blind", using fixture, spying on network

Tips & Tricks

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

The Funnel

Features vs Docs

⚖️

Improve the Docs: Personas

First time visitor

First time user

Repeat user

Paying customer

Promoter

🔑 Visitor / User should always find the right answer

First time visitor

What does this tool / service do?

Will it work for me?

🔑 Visitor / User should always find the right answer

First time user

How do I do X?

How do I do Y?

What do I do for error A?

First time user

🔑 Visitor / User should always find the right answer

Repeat user

How do I do X and Y and Z?

What do I do for error A, B, C?

How do I run using ...?

How do I learn more?

Repeat user

🔑 Visitor / User should always find the right answer

Paying customer

same as repeat user +

Is there training?

I got stuck, is there help?

Paying customer

🔑 Visitor / User should always find the right answer

Promoter

When are the new releases?

What else can this tool do?

Can I suggest a feature?

Is there an email list or webinars?

Docs keep growing

How to measure how good your documentation is ... and start crying

How to measure how good your documentation is ... and start crying

Documentation miss percentage

Documentation miss percentage

< 50% write more docs!

≥ 50%

docs have a problem

You Want Users to Find Answers to Their Questions by Themselves

The documentation demands for different user personas are contradictory

Why Good Docs Are Hard?

First time visitor

First time user

Repeat user

Paying customer