Deployment Pipelines: Disproving the Big Bang
@sublimino
Delivery Engineer
News UK, Visa, British Gas
www.binarysludge.com
What is a deployment Pipeline?
"Automated manifestation of your process for getting software from version control into the hands of your users"
- Jez Humble
(wrote the book)
or
The code's journey between
your mind and production
What is the big bang?
WHAT IS THE BIG BANG?
-
Late integration of software components
-
De facto industry standard
-
Bringer of release paralysis and fear
-
Useful in small projects where integration is minimal
Who is this TALK appropriate for?
Any developer that desires
quality, maintainability and sanity
Big Bang Integration testing is Bad
BIG BANG INTEGRATION TESTING IS BAD
-
Problems may not emerge until the system is under stress
- Which may prompt another release
-
System integration testing is SLOW
- Database fixtures are expensive to load
- Network communication is slow
- Anything browser-driven is slow
-
Stuck with long feedback loop
- May have to wait hours or overnight for results
BOTTOM UP INTEGRATION TESTING IS GOOD
- Encourages layered testing (unit, integration, functional)
- Find and fix bugs more quickly
-
Preferable to top down
- API definitions and mocks not required
- Avoids implementation-locking stub-hell
-
Problems discovered while the developer is in the zone
- Problem still fresh in the developer's mind
- No context switch required
- Less distance to travel through logs
But really, any testing is better than none at all
-
Good test practice:
- Describe the problem to be solved
- Identify the results required
- Write a failing test
- Hack it green
- Repeat, refactor, rejoice
the build pipeline
A set of engineering practices that ensure build quality
(Often require a cultural shift)
HOW?
-
Implicit trust in development team
-
No more:
- Release managers
- Go/No-Go meetings
- Release-driven testing
it begins with
Continuous Integration
"Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
By integrating regularly, you can detect errors quickly, and locate them more easily."
- http://www.thoughtworks.com/continuous-integration
Continuous integration
Is compatible with branching and PRs
Discourages languishing branches
Relatively widely practiced
Hosted services are available...
Full-stack testing
All code must be
automatically testable
-
Test teams should never be bigger than development teams
-
Humans are fallible, testers more so
-
Machines are all that will be left one day
(Private methods are contentious)
-
There is no right answer. Be pragmatic
-
If code must be tested (business critical, complex, due for refactoring) but is not in scope, consider separation by refactoring into a new class (only relevant in some cases)
-
BUT NEVER TEST IMPLEMENTATION
testing 101
-
Slice the problem into its component parts
-
Test in isolation
-
Test only one unit or intent (may use one or more assertions)
-
No inter-test dependencies
-
Tests can be run in parallel and in any order
-
Never test the same unit twice
-
If it’s passed once, accept it!
testing 102
-
Test only a single behaviour of a function
-
Only test the interface of what’s under test
-
Tests should provide a form of real-use documentation
-
Trust that where tests overlap, the path between them is tested
testing gotchas
-
DO NOT try to test the whole stack for every use case
-
Full stack tests tend towards giant scripting escapades
- brittle to run
- difficult to maintain
- modularisation and reuse impractical
- can lock-in implementation
-
Instead...
- write overlapping layers of tests
- never test the same thing twice
- trust that a passing test covers the case
next: Deployment automation
-
Frees up sysadmins to do their actual work
- Increase devops bus factor
-
Allows deployments on any schedule
- Increases business confidence
the Deployment Pipeline is the evolution of the CI approach
Continuousness:
Delivery vs Deployment
-
Delivery gets the code to a production-ready state
- but pushing the code to production is still required
- as are user acceptance tests
-
Deployment removes any human interaction
- The machines are in control
- UAT is performed by actual users
recap: THE THREE CS
- Continuous Integration
- tests running on every commit
- Continuous Delivery
- software is proven to be production-ready every commit
- every commit is a potential release candidate
- Continuous Deployment
- if every release candidate is near production, deploy it
What is in the pipeline?
-
Full-stack, confidence-imbuing tests
-
Hands-free automated deployment
-
Deep application and system monitoring
-
Equally hands-free automated rollback
automated deployment
Deploy, as long as it's automated:
-
Blue/Green with hot standby
- Incremental rollout
- Policy-based routing
- Any provable bunch o' scripts
monitoring and Logging
So much good stuff available:
-
statsd
-
graphite
-
graphana
-
logstash
-
kibana
-
datadog
automated rollback
Intelligent response to metrics and alerts
Get out of jail...
Not free
howto
Start with tests
Build a pipeline that's reliable
Deploy to an environment
Monitor everything
Use KPIs for a cluster immune system
Roll back if KPIs exceed tolerances
Halt the pipeline, analysis/5 whys?
Write a failing test case before fixing it
Re-enable the pipeline
Sleep well at night
questions?
@sublimino
www.binarysludge.com
Deployment Pipelines: Disproving the Big Bang
By Andrew Martin
