Effective Deployments

Mike Sherov

Head of Bit Moving

Behance Team @ Adobe

Do you care about the website?

Effective deployments are quick AND safe

Quick Deployments

• Build tools must run quickly
• The queue mustn't be bottlenecked
• Deployers must be prepared to verify
• No one in the queue must roll back
• Deployer must be able to debug build failures effectively

Quick Deployments

As the build step is happening, you should...

• Create a written list of things you need to do to verify the change.
• Open up all of the dashboards you will need to look at.
• If you don't have standard checklist dashes open, create a bookmark folder for them.

Quick Deployments

Avoid rollbacks!

• Stage absolutely must not be the first time you have exercised your changes.
• A rollback from stage wastes about 15 minutes of time
• A rollback from prod wastes about 20 minutes of time

Quick Deployments

Debugging Build Step Failures

• Open logs from build, look for failure msgs
• Run the build step locally with full rebase
• Look at trends to detect recent flakiness
• After all that fails, hit up slack channels

Quick Deployments

Debugging Deploy Step Failures

• Hit up slack channels
• Don't accept answers you don't understand

Safe Deployments

• Ensure it's the right time to be deploying
• Verify use cases thoroughly before reaching PROD and once again in PROD
• Communicate the change appropriately
• Know where to look for trouble
• Don't mark as good you fully verify it as safe
• Hang out in slack channels for a bit after deploy
• Don't be afraid to emergency rollback / lockdown

Safe Deployments

Make sure it's the right time to be deploying

• Don't deploy non-urgent changes after 5PM
• Don't deploy big changes on a Friday
• Is there are a code freeze soon?

Safe Deployments

Make sure it works on your local first

• Do you need to add fixtures?
• Have you tested in multiple browsers?
• Is your local PHP the right version?
• Have you run tests/linters locally?
• Have you actually thought deeply about how to verify this change?
• Have you checked with design?

Safe Deployments

Communicate your change

• Have you told community what the change is and why? It's failure mode? It's benefit?
• Have you looped in design for design review?
• Have you looped in ops if the change can have stability / perf implications?

Safe Deployments

Using Logging

• Do you have the dashboards open?
• Do you know how to search logs?

Safe Deployments

Using NewRelic

• Do you have PHP Errors open?
• Do you have JS Errors open?
• DO you know how to drill into PHP errors?
• Do you know how to drill into Perf issues?

Safe Deployments

Let it simmer 5 minutes
before marking as good.

• Some issues take several minutes to surface on any dashboard... it's OK, take your time!
• Test out the fix 1 or 2 more times while you wait.
• Look into other error messages or dashboards while you wait!

Safe Deployments

Assume any issue that comes up after your deploy was from your deploy

• If someone is deploying behind you, and they start experiencing issues in PROD, assume it's your fault.
• If community raises an issue shortly after your deploy, think about how it could be your issue.

Safe Deployments

Emergency rollbacks / lockdowns

• If you already marked as good, a Moonbeam admin can use emergency rollbacks to deploy a previous deploy to PROD.
• If fix can't be quickly found, put Moonbeam into lockdown mode, this prevents anyone from building and overwriting your rollback.
• Ensure next PR that goes up is either a reversion of the offending PR or fixes the offending PR.

Safe Deployments

What it feels like

• You aren't nervous going to PROD
• You are spending less time debugging PROD issues
• Less firefights
• Less complaints from the millions of daily users

What Questions do You Have?