Effective Deployments
Mike Sherov
Head of Bit Moving
Behance Team @ Adobe
Do you care about the website?
Effective deployments are quick AND safe
Quick Deployments
- Build tools must run quickly
- The queue mustn't be bottlenecked
- Deployers must be prepared to verify
- No one in the queue must roll back
- Deployer must be able to debug build failures effectively
Quick Deployments
As the build step is happening, you should...
- Create a written list of things you need to do to verify the change.
- Open up all of the dashboards you will need to look at.
- If you don't have standard checklist dashes open, create a bookmark folder for them.
Quick Deployments
Avoid rollbacks!
- Stage absolutely must not be the first time you have exercised your changes.
- A rollback from stage wastes about 15 minutes of time
- A rollback from prod wastes about 20 minutes of time
Quick Deployments
Debugging Build Step Failures
- Open logs from build, look for failure msgs
- Run the build step locally with full rebase
- Look at trends to detect recent flakiness
- After all that fails, hit up slack channels
Quick Deployments
Debugging Deploy Step Failures
- Hit up slack channels
- Don't accept answers you don't understand
Safe Deployments
- Ensure it's the right time to be deploying
- Verify use cases thoroughly before reaching PROD and once again in PROD
- Communicate the change appropriately
- Know where to look for trouble
- Don't mark as good you fully verify it as safe
- Hang out in slack channels for a bit after deploy
- Don't be afraid to emergency rollback / lockdown
Safe Deployments
Make sure it's the right time to be deploying
- Don't deploy non-urgent changes after 5PM
- Don't deploy big changes on a Friday
- Is there are a code freeze soon?
Safe Deployments
Make sure it works on your local first
- Do you need to add fixtures?
- Have you tested in multiple browsers?
- Is your local PHP the right version?
- Have you run tests/linters locally?
- Have you actually thought deeply about how to verify this change?
- Have you checked with design?
Safe Deployments
Communicate your change
- Have you told community what the change is and why? It's failure mode? It's benefit?
- Have you looped in design for design review?
- Have you looped in ops if the change can have stability / perf implications?
Safe Deployments
Using Logging
- Do you have the dashboards open?
- Do you know how to search logs?
Safe Deployments
Using NewRelic
- Do you have PHP Errors open?
- Do you have JS Errors open?
- DO you know how to drill into PHP errors?
- Do you know how to drill into Perf issues?
Safe Deployments
Let it simmer 5 minutes
before marking as good.
- Some issues take several minutes to surface on any dashboard... it's OK, take your time!
- Test out the fix 1 or 2 more times while you wait.
- Look into other error messages or dashboards while you wait!
Safe Deployments
Assume any issue that comes up after your deploy was from your deploy
- If someone is deploying behind you, and they start experiencing issues in PROD, assume it's your fault.
- If community raises an issue shortly after your deploy, think about how it could be your issue.
Safe Deployments
Emergency rollbacks / lockdowns
- If you already marked as good, a Moonbeam admin can use emergency rollbacks to deploy a previous deploy to PROD.
- If fix can't be quickly found, put Moonbeam into lockdown mode, this prevents anyone from building and overwriting your rollback.
- Ensure next PR that goes up is either a reversion of the offending PR or fixes the offending PR.
Safe Deployments
What it feels like
- You aren't nervous going to PROD
- You are spending less time debugging PROD issues
- Less firefights
- Less complaints from the millions of daily users