Where did we start?

• Master trunk
• No tests
• No automation
• No confidence

Where did we land?

Source: https://puppet.com/resources/whitepaper/state-of-devops-report

Where do you land?

Immediate changes made

• Get code onto Github
• Setup TeamCity/Octopus
• Implement gitflow
• Start writing tests

What was good?

• Less deployment surprises
• Testing was bringing more confidence
• Quality was improved

What was bad?

• Gitflow introduced artificial gates
• Most develop -> master merges were rubber stamps
• It slowed down our lead time for changes / MTTR
• Single line changes had to go through hoops
• Cherry picking became a default workflow
• Lots of rebasing
• Big bang releases (scary)

What else was bad?

• "Me too" commits, don't wanna miss the boat
• Same people releasing every time
• Low confidence to do a release

Spot the people releasing

Next suite of changes

• Drop gitflow
• Create docker environments on PRs
• Buy decent build machines

PR process

• PR created
• TeamCity picks up and builds changes
• If a docker label is added to the PR
  • Azure automation queues up a docker build
  • TeamCity runs a docker build
  • Docker environment created for changes in isolation

PR closed

• Azure automation scripts triggered
  • Tear down docker database
  • Connect to docker VM (Remote PowerShell)
  • Stop and remove container
• TeamCity picks up changes to master
• Builds and auto-deploys master to Test environment

Build server upgrade

35% quicker builds across the board

Upgrade build server

Yes one of those dropped by 8 minutes per build

Tell people about it!

Docker?

Why docker?

• Greater isolation of changes
• Smoke tests deployments (uses same packages)
• Greater testing for QA team
• Faster feedback from external business units
• Lead time for changes / MTTR down to <1hr
• Quick, clean teardown

Added benefits

• Clearer understanding of required infrastructure
• Knowledge that deployment would have failed
• Great for throwaway POCs
• No gitflow = greater incidence of refactoring

Most importantly

Removed the need for a develop environment

Current docker dependency tree

Example output from Azure Automation

Creation of docker container

How we view CD

• Make releases painless, non events
• Increase quality and stability
• Stop doing work we don't want to do

Octopus Deploy

• Each major project has it's own build/deploy pipeline
• Master is automatically promoted to TEST on green build
• Deploying to production still has a gate (for now)

Set up

• Two channels (Bargain Box vs MFB)
• Variables largely controlled by variable sets
• Some development tasks (e.g. restore PROD db)
• Slack integration for failure feedback

The good

• "It just works"
• Not hard to create releases and promote to PROD

The bad

• Easy to miss projects as part of a coordinated deploy
• We still have orchestrated deployments (boo)
• We still have to click the buttons
• Octopus Deploy has some weird limitations
  • No scheduled triggers
  • Can't deploy "whole projects" easily
• Configuration not checked into source control

How confident are we?

Testing strategy

Proposed changes

• Regression Suite
• Ramp up unit tests
• Remove reliance on Integration tests