C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
* self-managed
NOT WORTH IT
caching!!!
caching!!!
when is it ready?
how to stop it?
caching!!!
when is it ready?
how to stop it?
in parallel?
version: 2
jobs:
test:
docker:
- image: cypress/base:10
steps:
- checkout
# restore folders with npm dependencies and Cypress binary
- restore_cache:
keys:
- cache-{{ checksum "package.json" }}
# install npm dependencies and Cypress binary
# if they were cached, this step is super quick
- run:
name: Install dependencies
command: npm ci
- run: npm run cy:verify
# save npm dependencies and Cypress binary for future runs
- save_cache:
key: cache-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
# start server before starting tests
- run:
command: npm start
background: true
- run: npm run e2e:record
workflows:
version: 2
build:
jobs:
- test
Docker image
Caching
Caching
Install
run tests
maybe start app
defaults: &defaults
working_directory: ~/app
docker:
- image: cypress/browsers:chrome67
version: 2
jobs:
build:
<<: *defaults
steps:
- checkout
# find compatible cache from previous build,
# it should have same dependencies installed from package.json checksum
- restore_cache:
keys:
- cache-{{ .Branch }}-{{ checksum "package.json" }}
- run:
name: Install Dependencies
command: npm ci
# run verify and then save cache.
# this ensures that the Cypress verified status is cached too
- run: npm run cy:verify
# save new cache folder if needed
- save_cache:
key: cache-{{ .Branch }}-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
- run: npm run types
- run: npm run stop-only
# all other test jobs will run AFTER this build job finishes
# to avoid reinstalling dependencies, we persist the source folder "app"
# and the Cypress binary to workspace, which is the fastest way
# for Circle jobs to pass files
- persist_to_workspace:
root: ~/
paths:
- app
- .cache/Cypress
4x-electron:
<<: *defaults
# tell CircleCI to execute this job on 4 machines simultaneously
parallelism: 4
steps:
- attach_workspace:
at: ~/
- run:
command: npm start
background: true
# runs Cypress test in load balancing (parallel) mode
# and groups them in Cypress Dashboard under name "4x-electron"
- run: npm run e2e:record -- --parallel --group $CIRCLE_JOB
workflows:
version: 2
# this workflow has 4 jobs to show case Cypress --parallel and --group flags
# "build" installs NPM dependencies so other jobs don't have to
# โ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
# โ "4x-electron" job load balances all specs across 4 CI machines
# โ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
build_and_test:
jobs:
- build
# this group "4x-electron" will load balance all specs
# across 4 CI machines
- 4x-electron:
requires:
- build
Parallel config is ... more complicated
- install + run jobs
- workspace
- parallel flags
# first, install Cypress, then run all tests (in parallel)
stages:
- build
- test
# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
npm_config_cache: "$CI_PROJECT_DIR/.npm"
CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"
# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
# this job installs NPM dependencies and Cypress
install:
image: cypress/base:10
stage: build
script:
- npm ci
- $(npm bin)/print-env CI
- npm run cy:verify
# all jobs that actually run tests can use the same definition
.job_template: &job
image: cypress/base:10
stage: test
script:
# print CI environment variables for reference
- $(npm bin)/print-env CI
# start the server in the background
- npm run start:ci &
# run Cypress test in load balancing mode, pass id to tie jobs together
- npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons
# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
<<: *job
electrons-2:
<<: *job
electrons-3:
<<: *job
electrons-4:
<<: *job
electrons-5:
<<: *job
pipeline {
agent {
// this image provides everything needed to run Cypress
docker {
image 'cypress/base:10'
}
}
stages {
// first stage installs node dependencies and Cypress binary
stage('build') {
steps {
// there a few default environment variables on Jenkins
// on local Jenkins machine (assuming port 8080) see
// http://localhost:8080/pipeline-syntax/globals#env
echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
sh 'npm ci'
sh 'npm run cy:verify'
}
}
stage('start local server') {
steps {
// start local server in the background
// we will shut it down in "post" command block
sh 'nohup npm start &'
}
}
// this tage runs end-to-end tests, and each agent uses the workspace
// from the previous stage
stage('cypress parallel tests') {
environment {
// we will be recordint test results and video on Cypress dashboard
// to record we need to set an environment variable
// we can load the record key variable from credentials store
// see https://jenkins.io/doc/book/using/using-credentials/
CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
// because parallel steps share the workspace they might race to delete
// screenshots and videos folders. Tell Cypress not to delete these folders
CYPRESS_trashAssetsBeforeRuns = 'false'
}
// https://jenkins.io/doc/book/pipeline/syntax/#parallel
parallel {
// start several test jobs in parallel, and they all
// will use Cypress Dashboard to load balance any found spec files
stage('tester A') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
// second tester runs the same command
stage('tester B') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
}
}
}
post {
// shutdown the server running in the background
always {
echo 'Stopping local server'
sh 'pkill -f http-server'
}
}
}
language: node_js
node_js:
# Node 10.3+ includes npm@6 which has good "npm ci" command
- 10.8
cache:
# cache both npm modules and Cypress binary
directories:
- ~/.npm
- ~/.cache
override:
- npm ci
- npm run cy:verify
defaults: &defaults
script:
# ## print all Travis environment variables for debugging
- $(npm bin)/print-env TRAVIS
- npm start -- --silent &
- npm run cy:run -- --record --parallel --group $STAGE_NAME
# after all tests finish running we need
# to kill all background jobs (like "npm start &")
- kill $(jobs -p) || true
jobs:
include:
# we have multiple jobs to execute using just a single stage
# but we can pass group name via environment variable to Cypress test runner
- stage: test
env:
- STAGE_NAME=1x-electron
<<: *defaults
# run tests in parallel by including several test jobs with same name variable
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
defaults: &defaults
script:
- npm start -- --silent &
- stage: test
<<: *defaults
Single file, no variables, no code isolation
version: 2.1
orbs:
cypress: cypress-io/cypress@1
workflows:
build:
jobs:
- cypress/run
version: 2.1
orbs:
cypress: cypress-io/cypress@1
workflows:
build:
jobs:
- cypress/run
Reusable YAML from Orb registry
version: 2.1
orbs:
cypress: cypress-io/cypress@1
workflows:
build:
jobs:
- cypress/run
Substitute YAML from the orb
$ circleci config process circle.yml
version: 2.1
orbs:
cypress: cypress-io/cypress@1
workflows:
build:
jobs:
- cypress/run
CircleCI Orbs: text expansions + reusable shared configs
- cypress/install:
build: 'npm run build'
- cypress/run:
requires:
- cypress/install
record: true
parallel: true
parallelism: 4
Control behavior using parameters
"build" parameter of "install" job
- cypress/install:
build: 'npm run build'
- cypress/run:
requires:
- cypress/install
record: true
parallel: true
parallelism: 4
Control behavior using parameters
several parameter defined in the "run" job
- cypress/install:
build: 'npm run build'
- cypress/run:
requires:
- cypress/install
record: true
parallel: true
parallelism: 4
Control behavior using parameters
Allow the tool authors to write the specific CI config
name: GitHub Actions Demo
on: [push]
jobs:
Explore-GitHub-Actions:
runs-on: ubuntu-latest
steps:
- run: echo "๐ The job was automatically triggered by a ${{ github.event_name }} event."
- run: echo "๐ง This job is now running on a ${{ runner.os }} server hosted by GitHub!"
- run: echo "๐ The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
- name: Check out repository code
uses: actions/checkout@v3
- name: List files in the repository
run: |
ls ${{ github.workspace }}
GitHub CI system:
Github Actions
CI workflow file:
Github Actions
Reusable CI steps:
Github Actions
- name: Check out repository code
uses: actions/checkout@v3
Actions implemented by GitHub team (nothing extra-ordinary)
Hosted on https://github.com/actions
๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ๐คฏ
GitHub CI system:
Github Actions
CI workflow file:
Github Actions
Reusable CI steps:
Github Actions
GitHub CI system:
Actions CI
CI workflow file:
Workflow file
Reusable CI steps:
Github Actions
Reusable CI workflows:
Reusable workflows
name: GitHub Actions Demo
on: [push]
jobs:
Explore-GitHub-Actions:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
- name: Check out repository code
uses: actions/checkout@v3
name: 'Checkout'
description: 'Checkout a Git repository at a particular version'
inputs:
repository:
description: 'Repository name with owner. For example, actions/checkout'
default: ${{ github.repository }}
ref:
description: >
The branch, tag or SHA to checkout. When checking out the repository that
triggered a workflow, this defaults to the reference or SHA for that
event. Otherwise, uses the default branch.
token:
...
runs:
using: node16
main: dist/index.js
post: dist/index.js
steps:
# Reference a specific commit
- uses: actions/setup-node@74bc508
# Reference the major version of a release
- uses: actions/setup-node@v1
# Reference a minor version of a release
- uses: actions/setup-node@v1.2
# Reference a branch
- uses: actions/setup-node@master
name: main
on: [push]
jobs:
build-and-test:
runs-on: ubuntu-latest
name: Build and test
steps:
- uses: actions/checkout@v3
- uses: bahmutov/npm-install@v1
- run: npm t
list of steps: shell commands, 3rd party actions
name: main
on: [push]
jobs:
build-and-test:
runs-on: ubuntu-latest
name: Build and test
steps:
- name: Check out code ๐
uses: actions/checkout@v3
- name: Install dependencies ๐ฆ
uses: bahmutov/npm-install@v1
- name: Run tests ๐งช
run: npm t
Tip: give each step a good name
- uses: bahmutov/npm-install@v1
with:
working-directory: app1
# Do not detect and use a lock file
- uses: bahmutov/npm-install@v1
with:
useLockFile: false
# Use a custom install command
- uses: bahmutov/npm-install@v1
with:
install-command: yarn --frozen-lockfile --silent
# install prod dependencies only
- uses: bahmutov/npm-install@v1
env:
NODE_ENV: production
# install with some token
- uses: bahmutov/npm-install@v1
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
https://github.com/<org name>/<repo name>/settings/secrets/actions
{
"scripts": {
"build": "ncc build"
},
"dependencies": {
"@actions/core": "^1.10.0",
"@actions/exec": "^1.0.1",
"@actions/github": "^2.2.0",
"@actions/io": "^1.0.1",
"@actions/tool-cache": "^1.1.2",
"uuid": "^3.3.3"
},
"devDependencies": {
"@zeit/ncc": "^0.20.5"
}
}
A reusable action must be just a single JavaScript file that is ready to go*
* yes, you can make any action using Docker container, but who cares?!
runs:
using: node16
main: dist/index.js
post: dist/index.js
Making a JS action
- uses: actions/checkout@v3
# pick the Node version to use and install it
# https://github.com/actions/setup-node
- uses: actions/setup-node@v3
with:
node-version: 16
- uses: bahmutov/npm-install@v1
Design principles: have each action do 1 thing well
changed files
console.log(`::set-output name=CircleCIWorkflowUrl::${url}`)
A job can output values by simply printing with "::set-output" prefix
- name: Trigger the deployment
run: npx trigger-circleci-pipeline
id: trigger
- name: Print the workflow URL
run: echo Workflow URL ${{ steps.trigger.outputs.CircleCIWorkflowUrl }}
const url = getWebAppUrl(w)
if (process.env.GITHUB_STEP_SUMMARY) {
const summary = `CircleCI workflow ${w.name} URL: ${url}\n`
writeFileSync(process.env.GITHUB_STEP_SUMMARY, summary, {
flag: 'a+',
})
}
append to process.env.GITHUB_STEP_SUMMARY file
- uses: cypress-io/github-action@v4
- uses: actions/upload-artifact@v2
if: failure()
with:
name: cypress-screenshots
path: cypress/screenshots
- uses: actions/upload-artifact@v2
if: always()
with:
name: cypress-videos
path: cypress/videos
jobs:
node16:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 16
- uses: bahmutov/npm-install@v1
- run: npm test
node18:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- uses: bahmutov/npm-install@v1
- run: npm test
in parallel
jobs:
unit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: bahmutov/npm-install@v1
- run: npm test
e2e:
needs: unit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: cypress-io/github-action@v3
serial
install, caching, running E2E tests
jobs:
test:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: cypress-io/github-action@v4
test-cypress-v9:
...
release:
needs: [test, test-cypress-v9]
runs-on: ubuntu-20.04
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- run: npm install semantic-release
- name: Semantic Release ๐
uses: cycjimmy/semantic-release-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
Workflow execution
Workflow summary
jobs:
test:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: cypress-io/github-action@v4
test-cypress-v9:
...
release:
needs: [test, test-cypress-v9]
runs-on: ubuntu-20.04
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- run: npm install semantic-release
- name: Semantic Release ๐
uses: cycjimmy/semantic-release-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
Token for this repo only
created automagically ๐ช
name: badges
on:
push:
# update README badge only if the README file changes
# or if the package.json file changes, or this file changes
branches:
- master
paths:
- README.md
- package.json
- .github/workflows/badges.yml
schedule:
# update badges every night
# because we have a few badges that are linked
# to the external repositories
- cron: '0 4 * * *'
jobs:
badges:
name: Badges
runs-on: ubuntu-20.04
steps:
- name: Checkout ๐
uses: actions/checkout@v3
- name: Update version badges ๐ท
run: npx -p dependency-version-badge update-badge cypress
- name: Commit any changed files ๐พ
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Updated badges
branch: master
file_pattern: README.md
The "badged" workflow updated the README.md and pushed the change
Note: commits / events from GitHub Actions CI cannot trigger other GH Actions workflows to avoid infinite loops
name: Prettier
on: [push]
jobs:
build:
name: Prettier
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm run format
- name: Commit any changed files ๐พ
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Formatted the code
file_pattern: 'src/**/*.*'
# after checking out
# and testing it, building the site, etc
- name: Deploy ๐
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./public
- name: Set commit status ๐ซ
run: |
curl --request POST \
--url https://api.github.com/repos/${{ github.repository }}/statuses/${{ github.sha }} \
--header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
--header 'content-type: application/json' \
--data '{
"state": "success",
"description": "REST commit status",
"context": "a test"
}'
Details about the current repo and source reference
Main large objects
Example github context
- run: echo SHA ${{ github.sha }}
name: Parallel tests
on: [push]
jobs:
cypress-run:
strategy:
matrix:
machines: [1, 2, 3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
Matrices of jobs
name: Parallel tests
on: [push]
jobs:
cypress-run:
strategy:
matrix:
machines: [1, 2, 3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
Matrices of jobs
parallel-runs-across-platforms:
strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
machines: [1, 2]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
group: Parallel 2x on ${{ matrix.os }}
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
parallel-runs-across-platforms:
strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
machines: [1, 2]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
group: Parallel 2x on ${{ matrix.os }}
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
recorded parallel runs
Trigger events
name: The best workflow
on:
push
pull_request
schedule
on:
pull_request:
types: [opened]
jobs:
pr-labeler:
runs-on: ubuntu-latest
steps:
- uses: TimonVS/pr-labeler-action@v3
with:
configuration-path: .github/pr-labeler.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Labeled PRs based on branch names
Labeled PRs based on branch names
Trigger workflow on PR edit
name: pr
on:
pull_request:
types:
- edited
jobs:
rerun-tests:
steps:
- uses: bahmutov/should-run-github-action@v1
id: check-pr
env:
GITHUB_EVENT: ${{ toJson(github.event) }}
- if: ${{ steps.check-pr.outputs.shouldRun }}
run: npm test
Trigger workflow on PR edit
# .github/PULL_REQUEST_TEMPLATE.md
To re-run the tests, pick the tags above then click the checkbox below
- [ ] re-run the tests
name: pr
on:
pull_request:
types:
- edited
jobs:
rerun-tests:
steps:
- uses: bahmutov/should-run-github-action@v1
id: check-pr
env:
GITHUB_EVENT: ${{ toJson(github.event) }}
- if: ${{ steps.check-pr.outputs.shouldRun }}
run: npm test
Trigger workflow on PR edit
Manual triggers and UI
name: coupon tests
on:
# run these tests either manually or on schedule
workflow_dispatch:
description: Trigger coupon web tests
schedule:
# run these tests every day
- cron: '0 23 * * *'
jobs:
...
Manual dispatch parameters!
name: trigger-tests
on:
workflow_dispatch:
inputs:
# these inputs will be shown to the user on GitHub Actions page
# and the user can simply check off the tags to run
# NOTE: GitHub Action workflows have ten inputs max
sanity:
description: Run the tests tagged "@sanity", small set
required: false
type: boolean
all:
description: Run all the tests
required: false
type: boolean
# tests for individual features - the user will need to type the tags
# comma-separated in order to avoid hitting then ten workflow input limit
testTags:
description: |
Other test tags to run, comma-separated. Includes @profile, @signup,
...
# a few other utility params
testUrl:
description: The base URL to run tests against
required: false
type: string
machines:
description: Number of machines to use
required: false
type: integer
default: 8
...
Manual dispatch parameters!
- run: |
# collect all input parameters into one string
TAGS=
if [[ "${{ github.event.inputs.sanity }}" == "true" ]]; then
TAGS="@sanity"
fi
if [[ "${{ github.event.inputs.testTags }}" != "" ]]; then
TAGS="$TAGS,${{ github.event.inputs.testTags }}"
fi
echo "Collected tags: $TAGS"
echo "Number of machines to use: ${{ github.event.inputs.machines }}"
...
trigger tests
Manual dispatch parameters!
๐ฏ Any engineer can run end-to-end tests against any preview environment and observe the results
3:01am
name: hint
# run the workflow every morning
# or when we trigger the workflow manually
on:
workflow_dispatch:
inputs:
email:
description: Email to send the hint to
type: string
default: ''
required: false
hints:
description: Number of letters to reveal
type: integer
default: 1
schedule:
# UTC time
- cron: '0 7 * * *'
jobs:
hint:
runs-on: ubuntu-20.04
steps:
- name: Checkout ๐
uses: actions/checkout@v3
- name: Cypress run ๐งช
# https://github.com/cypress-io/github-action
uses: cypress-io/github-action@v3
with:
spec: 'cypress/integration/email-hint.js'
env: 'hints=${{ github.event.inputs.hints }}'
env:
SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
SENDGRID_FROM: ${{ secrets.SENDGRID_FROM }}
WORDLE_HINT_EMAIL: ${{ github.event.inputs.email || secrets.WORDLE_HINT_EMAIL }}
Reusable workflows
name: End-to-end tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v3
# Install NPM dependencies, cache them correctly
# and run all Cypress tests
- name: Cypress run
uses: cypress-io/github-action@v4
Same thing again and again
Reusable workflows
name: Parallel Cypress Tests
on: [push]
jobs:
test:
name: Cypress run
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
containers: [1, 2, 3]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Cypress run
uses: cypress-io/github-action@v4
with:
record: true
parallel: true
group: 'Actions example'
env:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ughh, syntax is powerful but verbose
Reusable workflows
name: ci
on: [push]
jobs:
test:
uses: bahmutov/cypress-workflows/.github/workflows/standard.yml@v1
single testing machine
name: ci
on: [push]
jobs:
test:
uses: bahmutov/cypress-workflows/.github/workflows/parallel.yml@v1
with:
n: 3
secrets:
recordKey: ${{ secrets.CYPRESS_RECORD_KEY }}
3 testing machines in parallel
Reusable workflows
My workflow A
My workflow B
Reusable
workflow X
GH Action 1
GH Action 2
GH Action 3
Isolation and reuse for your GitHub workflows and actions