Using End-to-end Tests as Documentation

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

survival is possible* but we need to act now

  • change your life
  • join an organization

rebellion.global          350.org

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional

(these slides)

cypress.tips

Agenda

  • App screenshots 🖼

  • Tests in Markdown 📝

  • How to have 100s of examples 

  • Current work

    • cypress-movie 🎥

  • keeping up to date

  • cypress-book

Let's Make a Web App

Markdown README.md

Is this still accurate?

Keep Screenshot Up-to-date

  • Once in a while take a screenshot and commit it
  • After each major release
  • How many senior engineers does it take to keep your documentation up-to-date and correct?
  • How much does it cost?
  • How much does it cost to have customers confused? Frustrated? Angry?

Keep Documentation Up-to-date

Tip: describe the screenshot

README.md

Replace Senior Engineers

with test automation

describe('Mailto', () => {
  it('screenshot', () => {
    cy.visit('/'); // localhost:3000
    cy.get('#to').type('me');
    cy.get('#subject').type('Date night');
    cy.get('#body').type('Have plans?');
    cy.get('#output').should(
      'have.text',
      'mailto:me?subject=Date%20night&body=Have%20plans%3F',
    );
    cy.get('#mailto').screenshot('mailto');
  });
});

Replace Senior Engineers

with test automation

describe('Mailto', () => {
  it('screenshot', () => {
    cy.visit('/'); // localhost:3000
    cy.get('#to').type('me');
    cy.get('#subject').type('Date night');
    cy.get('#body').type('Have plans?');
    cy.get('#output').should(
      'have.text',
      'mailto:me?subject=Date%20night&body=Have%20plans%3F',
    );
    cy.get('#mailto').screenshot('mailto');
  });
});

README.md

our test

manual update never works long-term

Test screenshot

README image

  • when running on CI
  • if the new image is different

Update screenshots only from CI

  • controlled environment
  • repeatable
  • can run on every commit or schedule
  • fast
    • update N screenshots
    • transform every image

cypress-book

Utility for updating screenshots from Cypress tests

$ npm i -D cypress-book
+ cypress-book@1.5.0
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      DEBUG: cypress-book
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # use https://github.com/cypress-io/github-action
      - name: Run tests 🧪
        uses: cypress-io/github-action@v2
        with:
          start: npm run dev

      - name: Copy social image
        run: |
          npx copy-image \
          -s cypress/screenshots/spec.js/mailto.png \
          -o media/demo.png

      # now let's see any changed files
      - name: See changed files 👀
        run: git status

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated screenshots
          file_pattern: '*.png'

.github/workflows/ci.yml

changed image is committed to the repository

changed image is committed to the repository

The tested README screenshot

The tested README screenshot

Version badge

name: badges
on:
  push:
    # update README badge only if the README file changes
    # or if the package.json file changes, or this file changes
    branches:
      - master
    paths:
      - README.md
      - package.json
      - .github/workflows/badges.yml
  schedule:
    # update badges every night
    # because we have a few badges that are linked
    # to the external repositories
    - cron: '0 3 * * *'

jobs:
  badges:
    name: Badges
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # using https://github.com/bahmutov/dependency-version-badge
      - name: Update version badges 🏷
        run: npx -p dependency-version-badge update-badge cypress

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated badges
          branch: master
          file_pattern: README.md

.github/workflows/badges.yml

Badges communicate

  • The project is trustworthy (CI tests pass)
  • The project is up-to-date

Updating screenshots

vs

Visual testing

  • Controlled enviroment (similar)
  • Replaces the screenshot IF there is a difference (opposite)

Tip: View Image Diffs on GitHub

added image

Tip: View Image Diffs on GitHub

The image is used here

The test code

making the image

is here

Can we keep these two related things close?

README.md as a spec file

# mailto ![cypress version](https://img.shields.io/badge/cypress-6.8.0-brightgreen)

![Demo image](./media/demo.png)

<!-- fiddle Mailto screenshot -->
```js
cy.visit('/'); // localhost:3000
cy.get('#to').type('me');
cy.get('#subject').type('Date night');
cy.get('#body').type('Have plans?');
cy.get('#output').should(
  'have.text',
  'mailto:me?subject=Date%20night&body=Have%20plans%3F',
);
cy.get('#mailto').screenshot('mailto');
```
<!-- fiddle-end -->

README.md

const mdPreprocessor = require('cypress-markdown-preprocessor');
module.exports = (on) => {
  on('file:preprocessor', mdPreprocessor);
};

cypress/plugins/index.js

README.md as a spec file

{
  "testFiles": "README.md",
  "integrationFolder": "."
}

cypress.json

Hide the test in the README

Tests should be readable

We can walk the user through the app using the Markdown + tests + screenshots

  • README
  • generated static documentation

README tests ⏩ screenshots ⏩ static project site

Cypress

VuePress

Markdown is 👑

A Better (Tested) Conference Talk Page

  • Index of the talk with direct links to the individual sections
  • Scrapable and searchable
  • Can be linked from other pages

Tests should be closer to their targets

Webinars + Conference Talks

@bahmutov on Twitter

gleb.bahmutov@gmail.com

If you have ideas how to include them better in the documentation, let me know

Let's Give Users 100s of Examples

Keeping Examples Correct and Up-to-date

The Current Approach

  1. Update the live HTML
  2. Update the spec file cypress/integration/querying.spec.js
  3. Copy updated test code into HTML page

Editing N places to update 1 test example ☹️

  • hard to remember
  • slow iteration

A Better (Tested) Cypress Examples Site

  • Single source of truth: Markdown
  • Generate the static site

If I see a question on a forum, GitHub, Twitter, etc

How do I toggle a checkbox?

App HTML

Test code

Markdown file

Markdown file

Markdown file

HTML code block

is mounted as live app

Markdown file

JS code block

runs as a test

Markdown spec also becomes a static docs page

Every example is scraped into the docs index

 💡Tip: Teach Users to Search

 💡Tip: Teach Users to Search

Presentations about documentation search

Documentation makes or breaks projects

Focus on the user

Remember: Are you selling the drills and not the holes

You are not there with the user

But your documentation is.

Current Work

  • screenshots ✅

  • test videos 🤔

Video of the entire spec file test run

A picture is worth a 1000 words

A 30fps 10 second video is worth 300,000 words

🖥 Tests are Demos

  • Cut the movie into individual clips
  • Slow down test steps
  • Add text overlays, arrows, highlights

ideas are welcome

Use Markdown to document

Conclusions

Embed tests in Markdown

Generate images and videos from tests

Thank you 👏

Gleb Bahmutov

Distinguished Engineer

Cypress.io 

@bahmutov