How to 1/x: Fast and Stable tests for sanity and profit

RUG::B Sep 2018, by @zalesz

@zalesz

RUG::B Sep 2018, by @zalesz

Development Lead at

Agenda

What's the problem?
Make it faster
Track and annihilate randoms

RUG::B Sep 2018, by @zalesz

What's

The

Problem?

RUG::B Sep 2018, by @zalesz

Oh no, not again..

RUG::B Sep 2018, by @zalesz

Couple restarts later..

... now we can deploy and go home

RUG::B Sep 2018, by @zalesz

Long feedback loops

What I was doing hour ago?

Wait for restart to deploy

Our test suite is basically useless

RUG::B Sep 2018, by @zalesz

Need

for

speed!

RUG::B Sep 2018, by @zalesz

Enter knapsack gem

RUG::B Sep 2018, by @zalesz

Installation

# Add gem in Gemfile
group :test, :development do
  gem "knapsack"
end

# Tasks in Rakefile
Knapsack.load_tasks if defined?(Knapsack)

# Setup in spec_helper.rb
if ENV["CI"]
  require "knapsack"
  Knapsack::Adapters::RSpecAdapter.bind
end

RUG::B Sep 2018, by @zalesz

Generate report on CI

# in .travis.yml replace default script with following
script: 
  - "KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec"

RUG::B Sep 2018, by @zalesz

After tests pass you should

Copy report which is rendered at the end of the tests in CI
Save it into your repository as knapsack_rspec_report.json

RUG::B Sep 2018, by @zalesz

After tests pass you should

# 3. Setup .travis.yml to run multiple jobs in parallel

env:
  global:
    - RAILS_ENV=test
    - CI_NODE_TOTAL=4 # total number of workers

  matrix:
    - CI_NODE_INDEX=0
    - CI_NODE_INDEX=1
    - CI_NODE_INDEX=2
    - CI_NODE_INDEX=3

script:
  - "bundle exec rake knapsack:rspec"

RUG::B Sep 2018, by @zalesz

Reap the benefits!

RUG::B Sep 2018, by @zalesz

Reap the benefits!

RUG::B Sep 2018, by @zalesz

Track and annihilate random failures!

RUG::B Sep 2018, by @zalesz

Find the failure

RUG::B Sep 2018, by @zalesz

Find the rspec seed

RUG::B Sep 2018, by @zalesz

Reproduce locally

$ CI_NODE_TOTAL=8 \
  CI_NODE_INDEX=5 \
  CI=true \
  bundle exec rake "knapsack:rspec[--seed 4049]"

RUG::B Sep 2018, by @zalesz

We need to go deeper!

CI_NODE_TOTAL=8 CI_NODE_INDEX=5 CI=true \
bundle exec rake "knapsack:rspec[--seed 4049 --bisect=verbose]"
# => 

Report specs:   # list of files selected by knapsack

Bisect started using options: "--seed 2463 --default-path spec -- 
spec/foo_spec.rb etc"

Running suite to find failures... (5 minutes 0 seconds)
 - Failing examples (3):
    - ./spec/controllers/api/v2/rooms_controller_spec.rb[1:1:1:2:2:1]
    - ./spec/features/attachment_spec.rb[1:2]
    - ./spec/features/workflow_spec.rb[1:1]
 - Non-failing examples (924):
    - rest of the files

Checking that failure(s) are order-dependent..
 - Running: rspec (50.83 seconds)
 - Failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-924
 - Multiple culprits detected - splitting candidates

Round 2: bisecting over non-failing examples 1-462
...

RUG::B Sep 2018, by @zalesz

Fix The Thing

And repeat :)

RUG::B Sep 2018, by @zalesz

Profit!

25 HOURS saved a day

~50' faster builds

~30 builds a day

RUG::B Sep 2018, by @zalesz

Profit!

Chill & rely on the

tests suite

Keep focused

Faster feedback

RUG::B Sep 2018, by @zalesz

Bonus: Cache CI dependencies

cache:
  bundler: true
  directories:
    - ~/bin

# install only test dependencies
bundler_args: "--without development production"

# use built in tools
addons:
  postgresql: '9.6'
  chrome: stable

before_install:
  - ./bin/ci_install_chromedriver

RUG::B Sep 2018, by @zalesz

Bonus: Cache CI dependencies

#!/bin/bash
set -Eeuo pipefail

VERSION="2.41"
NAME="chromedriver_${VERSION}"
STORAGE=https://chromedriver.storage.googleapis.com

install_chromedriver() {
  rm -f ~/bin/chromedriver*

  wget ${STORAGE}/${VERSION}/chromedriver_linux64.zip

  unzip chromedriver_linux64.zip
  rm chromedriver_linux64.zip
  mv -f chromedriver ~/bin/${NAME}
  chmod +x ~/bin/${NAME}

  ln -s ~/bin/${NAME} ~/bin/chromedriver
}

[ -e ~/bin/${NAME} ] || install_chromedriver

RUG::B Sep 2018, by @zalesz

Thanks!

Links:

RUG::B Sep 2018, by @zalesz