Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
In this presentation, I will show how simple the continuos integration can be with GitHub Actions. They are powerful, have generous limits for public repositories and can be easily reused.
https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end.
If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.
(take a deep breath)
Based on the blog post https://glebbahmutov.com/blog/trying-github-actions/
What are GitHub (GH) Actions
Hello world example
Fixing code formatting
Testing Node code
Caching
Writing your own action
Testing web apps using Cypress
Linux / Mac / Windows
parallelization
these slides
π¦ @bahmutov
Continuous integration (CI): every commit should be built and tested automatically in a clean environment.
Coding without CI is like never changing the oil in your car. It is just a matter of time before it blows up.
me (paraphrasing Justin James)
from https://cypress.slides.com/cypress-io/cypress-on-ci
Β
* self-managed
NOT WORTH IT
caching!!!
caching!!!
when is it ready?
how to stop it?
caching!!!
when is it ready?
how to stop it?
in parallel?
version: 2
jobs:
test:
docker:
- image: cypress/base:10
steps:
- checkout
# restore folders with npm dependencies and Cypress binary
- restore_cache:
keys:
- cache-{{ checksum "package.json" }}
# install npm dependencies and Cypress binary
# if they were cached, this step is super quick
- run:
name: Install dependencies
command: npm ci
- run: npm run cy:verify
# save npm dependencies and Cypress binary for future runs
- save_cache:
key: cache-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
# start server before starting tests
- run:
command: npm start
background: true
- run: npm run e2e:record
workflows:
version: 2
build:
jobs:
- test
Docker image
Caching
Caching
Install
run tests
maybe start app
defaults: &defaults
working_directory: ~/app
docker:
- image: cypress/browsers:chrome67
version: 2
jobs:
build:
<<: *defaults
steps:
- checkout
# find compatible cache from previous build,
# it should have same dependencies installed from package.json checksum
- restore_cache:
keys:
- cache-{{ .Branch }}-{{ checksum "package.json" }}
- run:
name: Install Dependencies
command: npm ci
# run verify and then save cache.
# this ensures that the Cypress verified status is cached too
- run: npm run cy:verify
# save new cache folder if needed
- save_cache:
key: cache-{{ .Branch }}-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
- run: npm run types
- run: npm run stop-only
# all other test jobs will run AFTER this build job finishes
# to avoid reinstalling dependencies, we persist the source folder "app"
# and the Cypress binary to workspace, which is the fastest way
# for Circle jobs to pass files
- persist_to_workspace:
root: ~/
paths:
- app
- .cache/Cypress
4x-electron:
<<: *defaults
# tell CircleCI to execute this job on 4 machines simultaneously
parallelism: 4
steps:
- attach_workspace:
at: ~/
- run:
command: npm start
background: true
# runs Cypress test in load balancing (parallel) mode
# and groups them in Cypress Dashboard under name "4x-electron"
- run: npm run e2e:record -- --parallel --group $CIRCLE_JOB
workflows:
version: 2
# this workflow has 4 jobs to show case Cypress --parallel and --group flags
# "build" installs NPM dependencies so other jobs don't have to
# β "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
# β "4x-electron" job load balances all specs across 4 CI machines
# β "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
build_and_test:
jobs:
- build
# this group "4x-electron" will load balance all specs
# across 4 CI machines
- 4x-electron:
requires:
- build
Parallel config is ... more complicated
- install + run jobs
- workspace
- parallel flags
# all jobs that actually run tests can use the same definition
job_template: &job_template
image: cypress/base:10
stage: test
script:
# print CI environment variables for reference
- $(npm bin)/print-env CI
# start the server in the background
- npm run start:ci &
# run Cypress test in load balancing mode
- npm run e2e:record -- --parallel --group "electrons on GitLab CI"
artifacts:
when: always
paths:
- cypress/videos/**/*.mp4
- cypress/screenshots/**/*.png
expire_in: 1 day
# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
<<: *job_template
electrons-2:
<<: *job_template
electrons-3:
<<: *job_template
electrons-4:
<<: *job_template
electrons-5:
<<: *job_template
YAML tips & tricks
# first, install Cypress, then run all tests (in parallel)
stages:
- build
- test
# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
npm_config_cache: "$CI_PROJECT_DIR/.npm"
CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"
# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
# this job installs NPM dependencies and Cypress
install:
image: cypress/base:10
stage: build
script:
- npm ci
- $(npm bin)/print-env CI
- npm run cy:verify
# all jobs that actually run tests can use the same definition
.job_template: &job
image: cypress/base:10
stage: test
script:
# print CI environment variables for reference
- $(npm bin)/print-env CI
# start the server in the background
- npm run start:ci &
# run Cypress test in load balancing mode, pass id to tie jobs together
- npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons
# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
<<: *job
electrons-2:
<<: *job
electrons-3:
<<: *job
electrons-4:
<<: *job
electrons-5:
<<: *job
pipeline {
agent {
// this image provides everything needed to run Cypress
docker {
image 'cypress/base:10'
}
}
stages {
// first stage installs node dependencies and Cypress binary
stage('build') {
steps {
// there a few default environment variables on Jenkins
// on local Jenkins machine (assuming port 8080) see
// http://localhost:8080/pipeline-syntax/globals#env
echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
sh 'npm ci'
sh 'npm run cy:verify'
}
}
stage('start local server') {
steps {
// start local server in the background
// we will shut it down in "post" command block
sh 'nohup npm start &'
}
}
// this tage runs end-to-end tests, and each agent uses the workspace
// from the previous stage
stage('cypress parallel tests') {
environment {
// we will be recordint test results and video on Cypress dashboard
// to record we need to set an environment variable
// we can load the record key variable from credentials store
// see https://jenkins.io/doc/book/using/using-credentials/
CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
// because parallel steps share the workspace they might race to delete
// screenshots and videos folders. Tell Cypress not to delete these folders
CYPRESS_trashAssetsBeforeRuns = 'false'
}
// https://jenkins.io/doc/book/pipeline/syntax/#parallel
parallel {
// start several test jobs in parallel, and they all
// will use Cypress Dashboard to load balance any found spec files
stage('tester A') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
// second tester runs the same command
stage('tester B') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
}
}
}
post {
// shutdown the server running in the background
always {
echo 'Stopping local server'
sh 'pkill -f http-server'
}
}
}
language: node_js
node_js:
# Node 10.3+ includes npm@6 which has good "npm ci" command
- 10.8
cache:
# cache both npm modules and Cypress binary
directories:
- ~/.npm
- ~/.cache
override:
- npm ci
- npm run cy:verify
defaults: &defaults
script:
# ## print all Travis environment variables for debugging
- $(npm bin)/print-env TRAVIS
- npm start -- --silent &
- npm run cy:run -- --record --parallel --group $STAGE_NAME
# after all tests finish running we need
# to kill all background jobs (like "npm start &")
- kill $(jobs -p) || true
jobs:
include:
# we have multiple jobs to execute using just a single stage
# but we can pass group name via environment variable to Cypress test runner
- stage: test
env:
- STAGE_NAME=1x-electron
<<: *defaults
# run tests in parallel by including several test jobs with same name variable
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
Β Β GitHub continuous integration
Β Β Β Define steps using 3rd party Docker / JavaScript / TypeScript
repo/ .github/ ISSUE_TEMPLATE.md PULL_REQUEST_TEMPLATE.md workflows/ hello.yml
.github/workflows/hello.yml
name: Hello
on: [push]
jobs:
hello:
runs-on: ubuntu-latest
steps:
- run: echo Hello World!
.github/workflows/hello.yml
name: Hello
on: [push]
jobs:
hello:
runs-on: ubuntu-latest
steps:
- run: echo Hello World!
.github/workflows/hello.yml
name: Hello
on: [push]
jobs:
hello:
runs-on: ubuntu-latest
steps:
- run: echo Hello World!
.github/workflows/hello.yml
name: Hello
on: [push]
jobs:
hello:
runs-on: ubuntu-latest
steps:
- run: echo Hello World!
https://github.com/<OWNER>/<REPOSITORY>/workflows/<WORKFLOW_NAME>/badge.svg?branch=<BRANCH>
# example
![cy-spok status](https://github.com/bahmutov/cy-spok/workflows/main/badge.svg?branch=master)
Markdown text
.github/workflows/docs.yml
name: Publish docs
on:
push:
branches:
- master
paths:
- 'docs/**'
jobs:
docs:
runs-on: ubuntu-latest
steps:
...
GH Actions are linked to the repo via GitHub Actions app installed automatically
automatic GITHUB_TOKEN
{
"scripts": {
"format": "prettier --write 'src/*.js'"
},
"devDependencies": {
"husky": "3.0.5",
"lint-staged": "9.2.5",
"prettier": "1.18.2"
},
"husky": {
"hooks": {
"pre-commit": "lint-staged"
}
},
"lint-staged": {
"*.js": ["prettier --write", "git add"]
}
}
but people (even I) forget
.github/workflows/ci.yml
name: Prettier
on: [push]
jobs:
build:
name: Prettier
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm run format
- run: git status
# commit any changed files
# https://github.com/mikeal/publish-to-github-action
- uses: mikeal/publish-to-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3rd party actions, each is
GH owner / repo @ commitish
.github/workflows/ci.yml
name: Prettier
on: [push]
jobs:
build:
name: Prettier
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm run format
- run: git status
# commit any changed files
# https://github.com/mikeal/publish-to-github-action
- uses: mikeal/publish-to-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3rd party action with passed GH token, can push code!
commit by action from workflow
Commit from GH Workflow
β οΈ there is no workflow triggered for the commit created from GH Action
Automated workflows cannot trigger other automated workflows (to avoid cycles?). Example: you can automatically open a PR but not close it.
https://github.com/mikeal/publish-to-github-action
#!/bin/sh
# check values
if [ -z "${GITHUB_TOKEN}" ]; then
echo "error: not found GITHUB_TOKEN"
exit 1
fi
# initialize git
remote_repo="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git config user.name "Automated Publisher"
git config user.email "publish-to-github-action@users.noreply.github.com"
git remote add publisher "${remote_repo}"
git show-ref # useful for debugging
git branch --verbose
# install lfs hooks
git lfs install
# publish any new files
git checkout master
git add -A
timestamp=$(date -u)
git commit -m "Automated publish: ${timestamp} ${GITHUB_SHA}" || exit 0
git pull --rebase publisher master
git push publisher master
#!/bin/sh
# check values
if [ -z "${GITHUB_TOKEN}" ]; then
echo "error: not found GITHUB_TOKEN"
exit 1
fi
# initialize git
remote_repo="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git config user.name "Automated Publisher"
git config user.email "publish-to-github-action@users.noreply.github.com"
git remote add publisher "${remote_repo}"
git show-ref # useful for debugging
git branch --verbose
# install lfs hooks
git lfs install
# publish any new files
git checkout master
git add -A
timestamp=$(date -u)
git commit -m "Automated publish: ${timestamp} ${GITHUB_SHA}" || exit 0
git pull --rebase publisher master
git push publisher master
.github/workflows/ci.yml
name: Prettier
on: [push]
jobs:
build:
name: Prettier
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm run format
- run: git status
# commit any changed files
# https://github.com/mikeal/publish-to-github-action
- uses: mikeal/publish-to-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3rd party GH action
GH Actions isolate complexity
3rd party GitHub Actions are a way to write CI code using JavaScript / TypeScript / anything
You
Tool's author
- uses: actions/checkout@v1
- name: Cache node modules
uses: actions/cache@v1
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: ${{ runner.os }}-node-
- run: npm ci
- run: npm test
ughh, I never remember these
const hasha = require('hasha')
// we are using dependency
// "cache": "github:cypress-io/github-actions-cache#8bec6cc"
const { restoreCache, saveCache } = require('cache/lib/index')
const useYarn = fs.existsSync('yarn.lock')
const lockFilename = useYarn ? 'yarn.lock' : 'package-lock.json'
const lockHash = hasha.fromFileSync(lockFilename)
const platformAndArch = `${process.platform}-${process.arch}`
// this is simplified code for clarity
// see action file index.js for full details
const NPM_CACHE = {
inputPath: '~/.npm', // or '~/.cache/yarn'
primaryKey: `npm-${platformAndArch}-${lockHash}`,
restoreKeys: `npm-${platformAndArch}-`
}
const restoreCachedNpm = () => {
console.log('trying to restore cached NPM modules')
return restoreCache(
NPM_CACHE.inputPath,
NPM_CACHE.primaryKey,
NPM_CACHE.restoreKeys
)
}
const saveCachedNpm = () => {
console.log('saving NPM modules')
return saveCache(NPM_CACHE.inputPath, NPM_CACHE.primaryKey)
}
restoreCachedNpm()
.then(npmCacheHit => {
console.log('npm cache hit', npmCacheHit)
return install().then(() => {
if (npmCacheHit) {
return
}
return saveCachedNpm()
})
})
.catch(error => {
console.log(error)
core.setFailed(error.message)
})
Write regular code to do things on CI
like restore and save cache folder
name: 'NPM or Yarn install with caching'
description: 'Install npm dependencies with caching'
author: 'Gleb Bahmutov'
runs:
using: 'node12'
main: 'dist/index.js'
branding:
color: 'yellow'
icon: 'command'
Declare your action in repo file
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm test
.github/workflows/ci.yml
name: Prettier
on: [push]
jobs:
build:
name: Prettier
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm run format
- run: git status
# commit any changed files
# https://github.com/mikeal/publish-to-github-action
- uses: mikeal/publish-to-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
# latest commit from a branch (DO NOT USE)
- uses: actions/setup-node@master
# Reference the major version of a release branch
# (I use this)
- uses: actions/setup-node@v1
# # Reference a minor version of a release (if exists)
- uses: actions/setup-node@v1.2
# Reference a specific commit (very strict)
- uses: actions/setup-node@74bc508
Each GH Action needs to be self-contained and ready to go.Β
If you use NPM deps, build a single bundle using @zeit/ncc
ncc build -o dist index.js
good example: https://github.com/bahmutov/npm-install
The GitHub Actions ToolKit provides a set of packages to make creating actions easier.
steps:
- uses: actions/checkout@v1
- uses: actions/setup-node@v1
with:
node-version: '10.x'
- run: npm install
- run: npm test
jobs:
build:
runs-on: ubuntu-16.04
strategy:
matrix:
node: [ '10', '8' ]
name: Node ${{ matrix.node }} sample
steps:
- uses: actions/checkout@v1
- uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node }}
- run: npm install
- run: npm test
creates 2 parallel isolated jobs
jobs:
build:
runs-on: ubuntu-16.04
strategy:
matrix:
node: [ '10', '8' ]
name: Node ${{ matrix.node }} sample
steps:
- uses: actions/checkout@v1
- uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node }}
- run: npm install
- run: npm test
name: End-to-end tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
(does npm install and caching, runs the tests correctly, can build app and start server and more)
name: Chome tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
browser: chrome
Use parameters to control behavior
name: Record test results
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
name: Parallel tests
on: [push]
jobs:
cypress-run:
strategy:
matrix:
machines: [1, 2, 3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
parallel-runs-across-platforms:
strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
machines: [1, 2]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
group: Parallel 2x on ${{ matrix.os }}
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
recorded parallel runs
http://tenminuteheroes.com/Β π¦ΈββοΈππ¦ΈββοΈ
By Gleb Bahmutov
In this presentation, I will show how simple continuous integration can be with GitHub Actions. They are powerful, have generous limits for public repositories, and can be easily reused. In particular, I will show a few use cases that can be of interest to any JavaScript or TypeScript developer: - How to run Prettier inside a GH Action and push any updated code back to the repository - How to correctly install Node dependencies with a single YAML line - How to run end-to-end Cypress tests without pulling your hair out. Video at https://www.youtube.com/watch?v=zhPqf5z_crc
JavaScript ninja, image processing expert, software quality fanatic