How to 1/x: Fast and Stable tests for sanity and profit
RUG::B Sep 2018, by @zalesz
@zalesz
RUG::B Sep 2018, by @zalesz
Development Lead at
Agenda
- What's the problem?
- Make it faster
- Track and annihilate randoms
RUG::B Sep 2018, by @zalesz
What's
The
Problem?
RUG::B Sep 2018, by @zalesz
Oh no, not again..
RUG::B Sep 2018, by @zalesz
Couple restarts later..
... now we can deploy and go home
RUG::B Sep 2018, by @zalesz
Long feedback loops
What I was doing hour ago?
Wait for restart to deploy
Our test suite is basically useless
RUG::B Sep 2018, by @zalesz
Need
for
speed!
RUG::B Sep 2018, by @zalesz
Enter knapsack gem
RUG::B Sep 2018, by @zalesz
Installation
# Add gem in Gemfile
group :test, :development do
gem "knapsack"
end# Tasks in Rakefile
Knapsack.load_tasks if defined?(Knapsack)# Setup in spec_helper.rb
if ENV["CI"]
require "knapsack"
Knapsack::Adapters::RSpecAdapter.bind
endRUG::B Sep 2018, by @zalesz
Generate report on CI
# in .travis.yml replace default script with following
script:
- "KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec"RUG::B Sep 2018, by @zalesz
After tests pass you should
- Copy report which is rendered at the end of the tests in CI
- Save it into your repository as knapsack_rspec_report.json
RUG::B Sep 2018, by @zalesz
After tests pass you should
# 3. Setup .travis.yml to run multiple jobs in parallel
env:
global:
- RAILS_ENV=test
- CI_NODE_TOTAL=4 # total number of workers
matrix:
- CI_NODE_INDEX=0
- CI_NODE_INDEX=1
- CI_NODE_INDEX=2
- CI_NODE_INDEX=3
script:
- "bundle exec rake knapsack:rspec"RUG::B Sep 2018, by @zalesz
Reap the benefits!
RUG::B Sep 2018, by @zalesz
Reap the benefits!
RUG::B Sep 2018, by @zalesz
Track and annihilate random failures!
RUG::B Sep 2018, by @zalesz
Find the failure
RUG::B Sep 2018, by @zalesz
Find the rspec seed
RUG::B Sep 2018, by @zalesz
Reproduce locally
$ CI_NODE_TOTAL=8 \
CI_NODE_INDEX=5 \
CI=true \
bundle exec rake "knapsack:rspec[--seed 4049]"RUG::B Sep 2018, by @zalesz
We need to go deeper!
CI_NODE_TOTAL=8 CI_NODE_INDEX=5 CI=true \
bundle exec rake "knapsack:rspec[--seed 4049 --bisect=verbose]"
# =>
Report specs: # list of files selected by knapsack
Bisect started using options: "--seed 2463 --default-path spec --
spec/foo_spec.rb etc"
Running suite to find failures... (5 minutes 0 seconds)
- Failing examples (3):
- ./spec/controllers/api/v2/rooms_controller_spec.rb[1:1:1:2:2:1]
- ./spec/features/attachment_spec.rb[1:2]
- ./spec/features/workflow_spec.rb[1:1]
- Non-failing examples (924):
- rest of the files
Checking that failure(s) are order-dependent..
- Running: rspec (50.83 seconds)
- Failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-924
- Multiple culprits detected - splitting candidates
Round 2: bisecting over non-failing examples 1-462
...RUG::B Sep 2018, by @zalesz
Fix The Thing
And repeat :)
RUG::B Sep 2018, by @zalesz
Profit!
25 HOURS saved a day
~50' faster builds
~30 builds a day
RUG::B Sep 2018, by @zalesz
Profit!
Chill & rely on the
tests suite
Keep focused
Faster feedback
RUG::B Sep 2018, by @zalesz
Bonus: Cache CI dependencies
cache:
bundler: true
directories:
- ~/bin
# install only test dependencies
bundler_args: "--without development production"
# use built in tools
addons:
postgresql: '9.6'
chrome: stable
before_install:
- ./bin/ci_install_chromedriverRUG::B Sep 2018, by @zalesz
Bonus: Cache CI dependencies
#!/bin/bash
set -Eeuo pipefail
VERSION="2.41"
NAME="chromedriver_${VERSION}"
STORAGE=https://chromedriver.storage.googleapis.com
install_chromedriver() {
rm -f ~/bin/chromedriver*
wget ${STORAGE}/${VERSION}/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
rm chromedriver_linux64.zip
mv -f chromedriver ~/bin/${NAME}
chmod +x ~/bin/${NAME}
ln -s ~/bin/${NAME} ~/bin/chromedriver
}
[ -e ~/bin/${NAME} ] || install_chromedriverRUG::B Sep 2018, by @zalesz
Thanks!
Links:
RUG::B Sep 2018, by @zalesz
How to 1/x: stable and faster tests for sanity and profit
By Leszek Zalewski
How to 1/x: stable and faster tests for sanity and profit
How to 1/x: stable and faster tests for sanity and profit
