Where did we land?

Source: https://puppet.com/resources/whitepaper/state-of-devops-report

Where do you land?

What was good?

• Less deployment surprises
• Testing was bringing more confidence
• Quality was improved

What was bad?

• Gitflow introduced artificial gates
• Most develop -> master merges were rubber stamps
• It slowed down our lead time for changes / MTTR
• Single line changes had to go through hoops
• Cherry picking became a default workflow
• Lots of rebasing
• Big bang releases (scary)

What else was bad?

• "Me too" commits, don't wanna miss the boat
• Same people releasing every time
• Low confidence to do a release

Spot the people releasing

PR process

• PR created
• TeamCity picks up and builds changes
• If a docker label is added to the PR
  • Azure automation queues up a docker build
  • TeamCity runs a docker build
  • Docker environment created for changes in isolation

PR closed

• Azure automation scripts triggered
  • Tear down docker database
  • Connect to docker VM (Remote PowerShell)
  • Stop and remove container
• TeamCity picks up changes to master
• Builds and auto-deploys master to Test environment

Upgrade build server

Yes one of those dropped by 8 minutes per build

Tell people about it!

Why docker?

• Greater isolation of changes
• Smoke tests deployments (uses same packages)
• Greater testing for QA team
• Faster feedback from external business units
• Lead time for changes / MTTR down to <1hr
• Quick, clean teardown

Added benefits

• Clearer understanding of required infrastructure
• Knowledge that deployment would have failed
• Great for throwaway POCs
• No gitflow = greater incidence of refactoring

Most importantly

Removed the need for a develop environment

Current docker dependency tree

Example output from Azure Automation

Creation of docker container

Octopus Deploy

• Each major project has it's own build/deploy pipeline
• Master is automatically promoted to TEST on green build
• Deploying to production still has a gate (for now)

Set up

• Two channels (Bargain Box vs MFB)
• Variables largely controlled by variable sets
• Some development tasks (e.g. restore PROD db)
• Slack integration for failure feedback

The good

• "It just works"
• Not hard to create releases and promote to PROD

The bad

• Easy to miss projects as part of a coordinated deploy
• We still have orchestrated deployments (boo)
• We still have to click the buttons
• Octopus Deploy has some weird limitations
  • No scheduled triggers
  • Can't deploy "whole projects" easily
• Configuration not checked into source control

Testing strategy

Proposed changes

• Regression Suite
• Ramp up unit tests
• Remove reliance on Integration tests