Transitioning Towards Continuous Delivery

A brief overview of how Continuous Delivery was adopted at Instagram and how it is/can be used at Bloomberg

Spencer Carver - October 14, 2016

Pretext

The information from Instagram in these slides is summarized from was Michael Gorven's talk "Continuous Deployment to Millions of Users 40 Times A Day" at Velocity NY on September 21, 2016.

If anything differs from the way it was originally presented, it is the fault of the current presenter's memory.

Instagram's Delivery Process
v1

1. Developer opens a PR

1

2. Code Review

2

4. Staging Verification

3

5. Canary Prod Verification

4

6. Staged Rollout (Cluster -> Region -> Fleet)

5

6

3. Test Suite

Manual

Automated

Instagram's Delivery Process
v2

1. Developer opens a PR

1

2. Code Review

2

4. Staging Verification

3

5. Canary Prod Verification

4

6. Staged Rollout (Cluster -> Region -> Fleet)

5

6

3. Test Suite

Manual

Automated

Instagram's Delivery Process
v2.5

1. Developer opens a PR

1

2. Code Review

2

4. Staging Verification

3

5. Canary Prod Verification

4

6. Staged Rollout (Cluster -> Region -> Fleet)

5

6

3. Test Suite

Manual

Automated

Instagram's Delivery Process
v3

1. Developer opens a PR

2

2. Code Review

1

4. Staging Verification

3

5. Canary Prod Verification

4

6. Staged Rollout (Cluster -> Region -> Fleet)

5

6

3. Test Suite

Manual

Automated

Instagram's Delivery Process
v3 (Pipelining)

Failure

Success

New deploy becomes base, grab next PR in queue

Old deploy still base, mark error and grab next PR in queue

Bloomberg Javelin Delivery Process

1. Developer opens a PR

1

2. Code Review

2

4. Staging Verification

3

5. Canary Prod Verification

4

6. Production Rollout

6

3. Test Suite

Manual

Automated

Build This

Stage This

Deploy This

Comparing Instagram v2, v3, and Javelin

1

2

3

4

5

6

2

1

3

4

5

6

1

2

3

4

6

Javelin

Instagram v2 (powered by human)

Instagram v3 (powered by Sauron)

My Takeaways

• Canary Prod
• Better Heathchecks / Automated Scan
• (Potentially) Change our workflow so that the tools we have are more fully able to pipeline deployments

• Most Importantly, remember this takes time. Instagram adopted it slowly over 2 years until all developers were comfortable with each step.