Site integrity

Unit and integration testing

Unit

• We test our components in isolation
• No connected dependencies are present
• Required external communication is blocked and responses simulated
• These tests can be executed without a network connection

Integration/E2E

• We run these in our build pipelines against the complete version of the site created with the latest code.
• We mimic production real world scenarios, verify data transmission
• We use tools to drive real browser engines, perform multipage visitor journeys

Basic goals

• Every design system component that includes javascript should have its own unit test along with its code
• Sites we deploy should have a set of end to end tests that run in the build/deploy pipeline to verify basic functionality

Latest browser landscape

npx playwright codegen https://covid19.ca.gov

Writing end to end tests using the playwright code generator

https://playwright.dev/

How do I run a test?

npm test

Checkout the commands in the scripts section of the package.json

Is shorthand for npm run test, and runs the script named test in the package.json scripts section

Running cypress tests

npm run test:server

https://www.cypress.io/

from the cannabis repository root

npx cypress open

We need to choose an integration testing tool

• Both reliable, ran for days every 15 mins in scheduled git actions
• Different out of the box approaches, command line(playwright) vs GUI (cypress)
• But both work headful and headlessly
• Different browser engine support (Only playwright supports Safari's webkit)
• Playwright evolving quickly
• Zakiya's excellent comparison notes
• More info in cannabis readme

Let's run some unit tests

• https://github.com/cagov/design-system
• Go to components/menu
• npm test

What is it doing?

• Unit tests based on the @open-wc/testing package
• Docs https://open-wc.org/docs/testing/testing-package/
• Test code we just ran is in test/index.js
• This unit test includes an axe accessibility review

commit and publish hooks

• Performs unit tests before the commit or publish operation
• Also runs code formatting with prettier
• Also runs lint checks with the open-wc eslint config

Code formatting

• We are using prettier, I heard from several team members that they like this tool. This is fine right?
• Configure prettier to run in your IDE if you like
• Build tools will run the prettier code formatting process
• This code format is never supposed to include any change that could modify functionality
• Use hide whitespace in github

Code linting

• In design-system repo
• In src/components/menu directory

npm run lint:eslint

npm run lint:eslint:fix

npm run lint

run eslint only, get a report of any issues

run both prettier and eslint

run eslint and have it fix anything it can safely

Testing is everybody's responsibility

• The dev team managing the test suites is wonderful, we decide the scope and get alerted to issues fast
• There should not be a single dev on the team who we rely on to write most of the tests
• Our test infrastructure will be most helpful if we all contribute and maintain it

What can go wrong with testing?

• Slow test suites

• Brittle tests that break with unrelated code changes

• Flaky tests

• Adding tests is too hard/not prioritized

• Centralized test runner security leaks

• Strict requirements like mandating 100% code coverage or to include a test with every code change cause problems

Writing tests can be fun!

• You are building reliability into the system
• You get immediate feedback when writing tests
• Tests are guides for open source code
• New contributors need tests to have confidence making changes

Tests are only a part of site integrity

What other areas do we need to address?