Я вижу что происходит: визуальное тестирование компонентов

Gleb Bahmutov

VP of Engineering

Cypress.io 

@bahmutov

kyiv

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

@bahmutov

survival is possible. but we need to act now

  • change your life
  • join an organization

@bahmutov

Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / React / Cycle.js / functional

(these slides)

https://slides.com/bahmutov

code at https://github.com/bahmutov/sudoku-qafest

live at https://sudoku-qafest.vercel.app/

code at https://github.com/bahmutov/sudoku-qafest

live at https://sudoku-qafest.vercel.app/

First cypress test

describe('Sudoku', () => {
  context('on mobile', () => {
    beforeEach(() => {
      cy.viewport(300, 600)
      cy.visit('/')
    })

    it('plays on mobile', () => {
      // on easy setting there are 45 filled cells at the start
      cy.get('.game__cell--filled').should('have.length', 45)
      cy.contains('.status__time', '00:00')
      cy.contains('.status__difficulty-select', 'Easy')
    })
  })
})

https://www.cypress.io/

describe('Sudoku', () => {
  context('on mobile', () => {
    beforeEach(() => {
      cy.viewport(300, 600)
      cy.visit('/')
    })

    it('plays on mobile', () => {
      // on easy setting there are 45 filled cells at the start
      cy.get('.game__cell--filled').should('have.length', 45)
      cy.contains('.status__time', '00:00')
      cy.contains('.status__difficulty-select', 'Easy')
    })
  })
})

First cypress test

https://www.cypress.io/

Continuous integration in the first 5 minutes

GitHub Actions, CircleCI, Netlify Build ...

name: push-tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v1
      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run ✅
        uses: cypress-io/github-action@v2
        with:
          start: npm start

GitHub Actions

https://glebbahmutov.com/blog/trying-github-actions/

https://glebbahmutov.com/blog/trying-github-actions/

https://glebbahmutov.com/blog/trying-github-actions/

- name: Cypress run ✅
  uses: cypress-io/github-action@v2
  with:
    start: npm start
    record: true
  env:
    # pass the Dashboard record key as an environment variable
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
    # pass GitHub token to allow accurately 
    # detecting a build vs a re-run build
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Store test artifacts

https://glebbahmutov.com/blog/trying-github-actions/

https://glebbahmutov.com/blog/trying-github-actions/

Caching is important

https://glebbahmutov.com/blog/trying-github-actions/

Preview deploys in the next 5 minutes

Vercel, Netlify ...

what about branches and pull requests?

.status__difficulty {
  /* position: relative; */
  top: 39px;
  left: 20px;
}

???

https://glebbahmutov.com/blog/develop-preview-test/

We deployed

We tested

https://glebbahmutov.com/blog/develop-preview-test/

name: deploy
on: [deployment_status] # instead of [push]
jobs:
  e2e:
    # only runs this job on successful deploy
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Print URL 🖨
        run: echo Testing URL ${{ github.event.deployment_status.target_url }}
      - name: Checkout 🛎
        uses: actions/checkout@v1
      - name: Run Cypress 🌲
        uses: cypress-io/github-action@v2
        with:
          record: true
          command-prefix: 'percy exec -- npx'
          group: deploy
        env:
          # url coming from Vercel deploy
          CYPRESS_BASE_URL: ${{ github.event.deployment_status.target_url }}
          # pass the Dashboard record key as an environment variable
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          # pass GitHub token to allow accurately
          # detecting a build vs a re-run build
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # Percy token for sending visual results
          PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}

Run E2E tests against the full preview deploy

https://glebbahmutov.com/blog/develop-preview-test/

Deploy started

Deploy finished

Does it look right?

👩‍💻 ✅

If I change this CSS (or class name or layout) just a little bit ...

🤖 🛑

@bahmutov

desktop

tablet

mobile

At every resolution?

Does it look right?

If I change this CSS (or class name or layout) just a little bit ...

👩‍💻 ✅ 🕰

🤖 🛑

Does it look the same?

🤖 ✅ ⏱

👩‍💻 🛑 🕰

If I change this CSS (or class name or layout) just a little bit ...

visual testing in the next 15 minutes

Percy, Applitools, Happo, open source ...

dynamic (random) data

clocks

animations and transitions

visual testing: potential problems

same test, no changes, is failing

cy.percySnapshot('mobile', {
  widths: [300],
  // hide all numbers when taking the snapshot
  percyCSS: `.game__cell--filled { opacity: 0; }`,
})

real visual diff

write more e2e Tests

Go through main user stories

use code coverage https://on.cypress.io/code-coverage

take visual snapshots

at major screens

Missing

playing the actual game

(because of the random board)

testing the components that build the game

in all possible states

Built from React Components

import React from 'react'
import { render } from 'react-dom'
import { App } from './App'

render(<App />, document.getElementById('root'))

index.js

import React from 'react'
import { Game } from './Game'
import './App.css'
import { SudokuProvider } from './context/SudokuContext'

export const App = () => {
  return (
    <SudokuProvider>
      <Game />
    </SudokuProvider>
  )
}

App.js

Top level component App

import React, { useState, useEffect } from 'react'
import moment from 'moment'
import { Header } from './components/layout/Header'
import { GameSection } from './components/layout/GameSection'
import { StatusSection } from './components/layout/StatusSection'
import { Footer } from './components/layout/Footer'
import { getUniqueSudoku } from './solver/UniqueSudoku'
import { useSudokuContext } from './context/SudokuContext'

export const Game = () => {
  ...
}

Game.js

Game component

return (
  <>
    <div className={overlay?"container blur":"container"}>
      <Header onClick={onClickNewGame}/>
      <div className="innercontainer">
        <GameSection
          onClick={(indexOfArray) => onClickCell(indexOfArray)}
        />
        <StatusSection
          onClickNumber={(number) => onClickNumber(number)}
          onChange={(e) => onChangeDifficulty(e)}
          onClickUndo={onClickUndo}
          onClickErase={onClickErase}
          onClickHint={onClickHint}
          onClickMistakesMode={onClickMistakesMode}
          onClickFastMode={onClickFastMode}
        />
      </div>
      <Footer />
    </div>
  </>
)

Game.js

Game component

Component Inputs

import React from 'react';
import { useSudokuContext } from '../context/SudokuContext';

/**
 * React component for the Number Selector in the Status Section.
 */
export const Numbers = (props) => {
  let { numberSelected } = useSudokuContext();
  return (
    <div className="status__numbers">
      {
        [1, 2, 3, 4, 5, 6, 7, 8, 9].map((number) => {
          if (numberSelected === number.toString()) {
            return (
              <div className="status__number status__number--selected"
                key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          } else {
            return (
              <div className="status__number" key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          }
        })
      }
    </div>
  )
}

Numbers.js

<Numbers onClickNumber={(number) => props.onClickNumber(number)} />

StatusSection.js

<Numbers .../>

props

context

user clicks

DOM

prop calls

Video Moving around the testing pyramid 

slides https://slides.com/bahmutov/assertjs

@bahmutov

check how component renders with different props / data

check how component behaves when you interact with it

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

yarn add -D cypress-react-unit-test

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

// cypress/support/index.js
require('cypress-react-unit-test/support')
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  return config
}
// cypress.json
{
  "experimentalComponentTesting": true,
  "componentFolder": "src"
}

@bahmutov

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

apply global styles

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

set the right structure

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

click a number

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>,
  )
  // use a single image snapshot after making sure
  // the component has been rendered into the DOM
  cy.get('.status__number').should('have.length', 9)
  cy.percySnapshot()
})

Numbers.spec.js

@bahmutov

visual component test

Assert the UI has updated before taking the snapshot

https://docs.percy.io/docs/cypress

it('shows selected number', () => {
  mount(
    <SudokuContext.Provider value={{ numberSelected: '4' }}>
      <div className="innercontainer">
        <section className="status">
          <Numbers />
        </section>
      </div>
    </SudokuContext.Provider>,
  )
  cy.contains('.status__number', '4').should(
    'have.class',
    'status__number--selected',
  )
  cy.percySnapshot()
})

Numbers.spec.js

@bahmutov

second visual component test

Deterministic Data: the Clock

What about the clock after 10 minutes?

What if the snapshot "catches" clock in transition?

Deterministic Data: the Clock

mount(
  <SudokuContext.Provider value={{ timeGameStarted: Cypress.moment() }}>
    <div className="innercontainer">
      <section className="status">
        <Timer />
      </section>
    </div>
  </SudokuContext.Provider>,
)
cy.contains('.status__time', '00:00')
cy.contains('.status__time', '00:01')
cy.contains('.status__time', '00:02')
cy.contains('.status__time', '00:03')
mount(
  <SudokuContext.Provider value={{ timeGameStarted: Cypress.moment() }}>
    <div className="innercontainer">
      <section className="status">
        <Timer />
      </section>
    </div>
  </SudokuContext.Provider>,
)
cy.contains('.status__time', '00:00')
cy.contains('.status__time', '00:01')
cy.contains('.status__time', '00:02')
cy.contains('.status__time', '00:03')

Deterministic Data: the Clock at 00:00

const now = Cypress.moment()
cy.clock() // freeze the clock
mount(
  <SudokuContext.Provider value={{ timeGameStarted: now }}>
    <div className="innercontainer">
      <section className="status">
        <Timer />
      </section>
    </div>
  </SudokuContext.Provider>,
)
cy.contains('.status__time', '00:00')
cy.percySnapshot()

@bahmutov

Deterministic Data: the Clock after 700s

const now = Cypress.moment()
cy.clock(now.clone().add(700, 'seconds').toDate())
mount(
  <SudokuContext.Provider value={{ timeGameStarted: now }}>
    <div className="innercontainer">
      <section className="status">
        <Timer />
      </section>
    </div>
  </SudokuContext.Provider>,
)
cy.contains('.status__time', '11:40')
cy.percySnapshot()

@bahmutov

component tests: any level

?

<App />

<Game />

<Header />

<GameSection />

<StatusSection />

<Footer />

<Timer />

<Difficulty />

<Numbers />

tested

tested

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').percySnapshot()
})

App.spec.js

@bahmutov

Why not the entire game?

component tests: any level

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.get('.container').percySnapshot()
})

App.spec.js

Because every time test runs, a new random board will be generated

component tests: any level

// App.js uses Game.js
// Game.js
import { getUniqueSudoku } from './solver/UniqueSudoku'
...
function _createNewGame(e) {
  let [temporaryInitArray, temporarySolvedArray] = getUniqueSudoku(difficulty, e);
  ...
}

// cypress/fixtures/init-array.json
["0", "0", "9", "0", "2", "0", "0", ...]
// cypress/fixtures/solved-array.json
["6", "7", "9", "3", "2", "8", "4", ...]

@bahmutov

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('mocks board creation', () => {
  // load JSON files using cy.fixture calls
  // https://on.cypress.io/fixture
  cy.fixture('init-array').then((initArray) => {
    cy.fixture('solved-array').then((solvedArray) => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([
        initArray,
        solvedArray,
      ])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell--filled').should('have.length', 45)
  // the visual snapshot will be the same
  cy.percySnapshot()
})

mock component methods

mock ES6 import

import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('mocks board creation', () => {
  // load JSON files using cy.fixture calls
  // https://on.cypress.io/fixture
  cy.fixture('init-array').then((initArray) => {
    cy.fixture('solved-array').then((solvedArray) => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([
        initArray,
        solvedArray,
      ])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell--filled').should('have.length', 45)
  // the visual snapshot will be the same
  cy.percySnapshot()
})

mock component methods

mock ES6 import

it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .percySnapshot()
})
it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container')
    .percySnapshot()
})

See What The Test Is Doing

it('plays to win', () => {
  // start with all but the first cell filled with solved array
  const almostSolved = [...solvedArray]
  // by setting entry to "0" we effectively clear the cell
  almostSolved[0] = '0'
  cy.stub(UniqueSudoku, 'getUniqueSudoku')
    .returns([almostSolved, solvedArray])
    .as('getUniqueSudoku')
  cy.clock()
  mount(<App />)
  cy.visualSnapshot('1 game is almost solved')

  // win the game
  cy.get('.game__cell').first().click()
  // use the known number to fill the first cell
  cy.contains('.status__number', solvedArray[0]).click()
  
  // winning message displayed
  cy.get('.overlay__text').should('be.visible')
  cy.visualSnapshot('2 game is solved')

  // clicking the overlay starts the new game
  cy.get('@getUniqueSudoku').should('have.been.calledOnce')
  cy.get('.overlay__text').click()
  cy.get('.overlay').should('not.be.visible')
  cy.get('@getUniqueSudoku').should('have.been.calledTwice')
})

play the full game via component test

it('plays to win', () => {
  // start with all but the first cell filled with solved array
  const almostSolved = [...solvedArray]
  // by setting entry to "0" we effectively clear the cell
  almostSolved[0] = '0'
  cy.stub(UniqueSudoku, 'getUniqueSudoku')
    .returns([almostSolved, solvedArray])
    .as('getUniqueSudoku')
  cy.clock()
  mount(<App />)
  cy.visualSnapshot('1 game is almost solved')

  // win the game
  cy.get('.game__cell').first().click()
  // use the known number to fill the first cell
  cy.contains('.status__number', solvedArray[0]).click()
  
  // winning message displayed
  cy.get('.overlay__text').should('be.visible')
  cy.visualSnapshot('2 game is solved')

  // clicking the overlay starts the new game
  cy.get('@getUniqueSudoku').should('have.been.calledOnce')
  cy.get('.overlay__text').click()
  cy.get('.overlay').should('not.be.visible')
  cy.get('@getUniqueSudoku').should('have.been.calledTwice')
})

play the full game via component test

visual testing tips

Always use an assertion before the visual snapshot command

// use a single image snapshot after making sure
// the component has been rendered into the DOM
cy.get('.status__number').should('have.length', 9)
cy.percySnapshot()
cy.get('.overlay__text').should('be.visible')
cy.percySnapshot('2 game is solved')

visual testing tips

Cypress.Commands.add('visualSnapshot', (maybeName) => {
  let snapshotTitle = cy.state('runnable').fullTitle()
  if (maybeName) {
    snapshotTitle = snapshotTitle + ' - ' + maybeName
  }
  cy.percySnapshot(snapshotTitle, {
    widths: [cy.state('viewportWidth')],
    minHeight: cy.state('viewportHeight'),
  })
})

Write a snapshot wrapper command for convenience

visual testing tips

Write a snapshot wrapper command for convenience

// single snapshot in the test
cy.visualSnapshot()

// several snapshots in the test
cy.visualSnapshot('1 game is almost solved')
// then later
cy.visualSnapshot('2 game is solved')

visual testing tips

Test components at different resolutions

const playGame = () => {
  ...
  cy.clock()
  mount(<App />)
  cy.visualSnapshot('1 game is almost solved')

  // win the game
  cy.get('.game__cell').first().click()
  // use the known number to fill the first cell
  cy.contains('.status__number', solvedArray[0]).click()

  // winning message displayed
  cy.get('.overlay__text').should('be.visible')
  cy.visualSnapshot('2 game is solved')
}

it is just JavaScript!

visual testing tips

Test components at different resolutions

// using different viewport resolutions run the same test
// https://on.cypress.io/viewport
const tablet = [660, 700]
const phone = [400, 700]

;[tablet, phone].forEach((resolution) => {
  it(`${resolution[0]}x${resolution[1]}`, () => {
    cy.viewport(...resolution)
    playGame()
  })
})

it is just JavaScript!

visual testing tips

Test components at different resolutions

// using different viewport resolutions run the same test
// https://on.cypress.io/viewport
const tablet = [660, 700]
const phone = [400, 700]

;[tablet, phone].forEach((resolution) => {
  it(`${resolution[0]}x${resolution[1]}`, () => {
    cy.viewport(...resolution)
    playGame()
  })
})

it is just JavaScript!

Final view of testing

Final view of testing

Final view of testing

Final view of testing

Gleb Bahmutov @bahmutov https://gleb.dev/