Deployment Pipelines: Disproving the Big Bang

@sublimino

Delivery Engineer

News UK, Visa, British Gas

www.binarysludge.com

What is a deployment Pipeline?

"Automated manifestation of your process for getting software from version control into the hands of your users"

- Jez Humble

(wrote the book)

or

The code's journey between

your mind and production

What is the big bang?

WHAT IS THE BIG BANG?

Late integration of software components
De facto industry standard
Bringer of release paralysis and fear
Useful in small projects where integration is minimal

Who is this TALK appropriate for?

Any developer that desires

quality, maintainability and sanity

Big Bang Integration testing is Bad

Isolating defects is hard - many logs, little granularity

Interface problems are not discovered until components are integrated

Business pressures often push this to the end of the development lifecycle

This leads to stressful development

BIG BANG INTEGRATION TESTING IS BAD

Problems may not emerge until the system is under stress

Which may prompt another release

System integration testing is SLOW

Database fixtures are expensive to load
Network communication is slow
Anything browser-driven is slow

Stuck with long feedback loop

May have to wait hours or overnight for results

BOTTOM UP INTEGRATION TESTING IS GOOD

Encourages layered testing (unit, integration, functional)

Find and fix bugs more quickly

Preferable to top down

API definitions and mocks not required
Avoids implementation-locking stub-hell

Problems discovered while the developer is in the zone

Problem still fresh in the developer's mind
No context switch required
Less distance to travel through logs

But really, any testing is better than none at all

Good test practice:

Describe the problem to be solved
Identify the results required
Write a failing test
Hack it green
Repeat, refactor, rejoice

the build pipeline

A set of engineering practices that ensure build quality

(Often require a cultural shift)

HOW?

Implicit trust in development team
No more:

Release managers
Go/No-Go meetings
Release-driven testing

it begins with

Continuous Integration

"Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

By integrating regularly, you can detect errors quickly, and locate them more easily."

- http://www.thoughtworks.com/continuous-integration

Continuous integration

Is compatible with branching and PRs

Discourages languishing branches

Relatively widely practiced

Hosted services are available...

http://www.infoq.com/research/ci-server

Full-stack testing

All code must be

automatically testable

Test teams should never be bigger than development teams
Humans are fallible, testers more so
Machines are all that will be left one day

(Private methods are contentious)

There is no right answer. Be pragmatic
If code must be tested (business critical, complex, due for refactoring) but is not in scope, consider separation by refactoring into a new class (only relevant in some cases)
BUT NEVER TEST IMPLEMENTATION

testing 101

Slice the problem into its component parts
Test in isolation
Test only one unit or intent (may use one or more assertions)
No inter-test dependencies
Tests can be run in parallel and in any order
Never test the same unit twice
If it’s passed once, accept it!

testing 102

Test only a single behaviour of a function
Only test the interface of what’s under test
Tests should provide a form of real-use documentation
Trust that where tests overlap, the path between them is tested

testing gotchas

DO NOT try to test the whole stack for every use case
Full stack tests tend towards giant scripting escapades

brittle to run
difficult to maintain
modularisation and reuse impractical
can lock-in implementation

Instead...

write overlapping layers of tests
never test the same thing twice
trust that a passing test covers the case

next: Deployment automation

Frees up sysadmins to do their actual work
Increase devops bus factor
Allows deployments on any schedule
Increases business confidence

the Deployment Pipeline is the evolution of the CI approach

Continuousness:

Delivery vs Deployment

Delivery gets the code to a production-ready state

but pushing the code to production is still required
as are user acceptance tests

Deployment removes any human interaction

The machines are in control
UAT is performed by actual users

recap: THE THREE CS

Continuous Integration

tests running on every commit

Continuous Delivery

software is proven to be production-ready every commit
every commit is a potential release candidate

Continuous Deployment

if every release candidate is near production, deploy it

What is in the pipeline?

Full-stack, confidence-imbuing tests
Hands-free automated deployment
Deep application and system monitoring
Equally hands-free automated rollback

automated deployment

Deploy, as long as it's automated:

Blue/Green with hot standby
Incremental rollout
Policy-based routing
Any provable bunch o' scripts

monitoring and Logging

So much good stuff available:

statsd
graphite
graphana
logstash
kibana
datadog

automated rollback

Intelligent response to metrics and alerts

Get out of jail...

Not free

howto

Start with tests

Build a pipeline that's reliable

Deploy to an environment

Monitor everything

Use KPIs for a cluster immune system

Roll back if KPIs exceed tolerances

Halt the pipeline, analysis/5 whys?

Write a failing test case before fixing it

Re-enable the pipeline

Sleep well at night

questions?

@sublimino