Using End-to-end Tests as Documentation

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • join an organization

rebellion.global          350.org

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

(these slides)

cypress.tips

Agenda

  • App screenshots 🖼

  • Tests in Markdown 📝

  • How to have 100s of examples 

  • Current work

    • cypress-movie 🎥

  • keeping up to date

  • cypress-book

Let's Make a Web App

Markdown README.md

Is this still accurate?

Keep Screenshot Up-to-date

  • Once in a while take a screenshot and commit it
  • After each major release
  • How many senior engineers does it take to keep your documentation up-to-date and correct?
  • How much does it cost?
  • How much does it cost to have customers confused? Frustrated? Angry?

Keep Documentation Up-to-date

Tip: describe the screenshot

README.md

Replace Senior Engineers

with test automation

describe('Mailto', () => {
  it('screenshot', () => {
    cy.visit('/'); // localhost:3000
    cy.get('#to').type('me');
    cy.get('#subject').type('Date night');
    cy.get('#body').type('Have plans?');
    cy.get('#output').should(
      'have.text',
      'mailto:me?subject=Date%20night&body=Have%20plans%3F',
    );
    cy.get('#mailto').screenshot('mailto');
  });
});

Replace Senior Engineers

with test automation

describe('Mailto', () => {
  it('screenshot', () => {
    cy.visit('/'); // localhost:3000
    cy.get('#to').type('me');
    cy.get('#subject').type('Date night');
    cy.get('#body').type('Have plans?');
    cy.get('#output').should(
      'have.text',
      'mailto:me?subject=Date%20night&body=Have%20plans%3F',
    );
    cy.get('#mailto').screenshot('mailto');
  });
});

README.md

our test

manual update never works long-term

Test screenshot

README image

  • when running on CI
  • if the new image is different

Update screenshots only from CI

  • controlled environment
  • repeatable
  • can run on every commit or schedule
  • fast
    • update N screenshots
    • transform every image

cypress-book

Utility for updating screenshots from Cypress tests

$ npm i -D cypress-book
+ cypress-book@1.5.0
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      DEBUG: cypress-book
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # use https://github.com/cypress-io/github-action
      - name: Run tests 🧪
        uses: cypress-io/github-action@v2
        with:
          start: npm run dev

      - name: Copy social image
        run: |
          npx copy-image \
          -s cypress/screenshots/spec.js/mailto.png \
          -o media/demo.png

      # now let's see any changed files
      - name: See changed files 👀
        run: git status

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated screenshots
          file_pattern: '*.png'

.github/workflows/ci.yml

changed image is committed to the repository

changed image is committed to the repository

The tested README screenshot

The tested README screenshot

Version badge

name: badges
on:
  push:
    # update README badge only if the README file changes
    # or if the package.json file changes, or this file changes
    branches:
      - master
    paths:
      - README.md
      - package.json
      - .github/workflows/badges.yml
  schedule:
    # update badges every night
    # because we have a few badges that are linked
    # to the external repositories
    - cron: '0 3 * * *'

jobs:
  badges:
    name: Badges
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # using https://github.com/bahmutov/dependency-version-badge
      - name: Update version badges 🏷
        run: npx -p dependency-version-badge update-badge cypress

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated badges
          branch: master
          file_pattern: README.md

.github/workflows/badges.yml

Badges communicate

  • The project is trustworthy (CI tests pass)
  • The project is up-to-date

Updating screenshots

vs

Visual testing

  • Controlled enviroment (similar)
  • Replaces the screenshot IF there is a difference (opposite)

Tip: View Image Diffs on GitHub

added image

Tip: View Image Diffs on GitHub

The image is used here

The test code

making the image

is here

Can we keep these two related things close?

README.md as a spec file

# mailto ![cypress version](https://img.shields.io/badge/cypress-6.8.0-brightgreen)

![Demo image](./media/demo.png)

<!-- fiddle Mailto screenshot -->
```js
cy.visit('/'); // localhost:3000
cy.get('#to').type('me');
cy.get('#subject').type('Date night');
cy.get('#body').type('Have plans?');
cy.get('#output').should(
  'have.text',
  'mailto:me?subject=Date%20night&body=Have%20plans%3F',
);
cy.get('#mailto').screenshot('mailto');
```
<!-- fiddle-end -->

README.md

const mdPreprocessor = require('cypress-markdown-preprocessor');
module.exports = (on) => {
  on('file:preprocessor', mdPreprocessor);
};

cypress/plugins/index.js

README.md as a spec file

{
  "testFiles": "README.md",
  "integrationFolder": "."
}

cypress.json

Hide the test in the README

Tests should be readable

We can walk the user through the app using the Markdown + tests + screenshots

  • README
  • generated static documentation

README tests ⏩ screenshots ⏩ static project site

Cypress

VuePress

Markdown is 👑

A Better (Tested) Conference Talk Page

  • Index of the talk with direct links to the individual sections
  • Scrapable and searchable
  • Can be linked from other pages

Tests should be closer to their targets

Webinars + Conference Talks

@bahmutov on Twitter

gleb.bahmutov@gmail.com

If you have ideas how to include them better in the documentation, let me know

Let's Give Users 100s of Examples

Keeping Examples Correct and Up-to-date

The Current Approach

  1. Update the live HTML
  2. Update the spec file cypress/integration/querying.spec.js
  3. Copy updated test code into HTML page

Editing N places to update 1 test example ☹️

  • hard to remember
  • slow iteration

A Better (Tested) Cypress Examples Site

  • Single source of truth: Markdown
  • Generate the static site

If I see a question on a forum, GitHub, Twitter, etc

How do I toggle a checkbox?

App HTML

Test code

Markdown file

Markdown file

Markdown file

HTML code block

is mounted as live app

Markdown file

JS code block

runs as a test

Markdown spec also becomes a static docs page

Every example is scraped into the docs index

 💡Tip: Teach Users to Search

 💡Tip: Teach Users to Search

Presentations about documentation search

Documentation makes or breaks projects

Focus on the user

Remember: Are you selling the drills and not the holes

You are not there with the user

But your documentation is.

Current Work

  • screenshots ✅

  • test videos 🤔

Video of the entire spec file test run

A picture is worth a 1000 words

A 30fps 10 second video is worth 300,000 words

🖥 Tests are Demos

  • Cut the movie into individual clips
  • Slow down test steps
  • Add text overlays, arrows, highlights

ideas are welcome

Use Markdown to document

Conclusions

Embed tests in Markdown

Generate images and videos from tests

Thank you 👏

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov

Using End-to-end Tests as Documentation

By Gleb Bahmutov

Using End-to-end Tests as Documentation

Keeping the documentation up-to-date with the web application is hard. The screenshots and the videos showing the user how to perform some task quickly fall out of sync with the latest design and logic changes. In this presentation, I will show how to use end-to-end tests to generate the documentation. By keeping the tests in sync with the application, and by running them on every commit, we will update the documentation, ensuring our users never get confused by the obsolete docs. Presented at TestingStage 2021, video at https://youtu.be/H9VqsTZ9NME

  • 2,914