Roll Forward,
Roll Back,
Cancel the Release
About Me
- I help people take Vacations!
- Principal Engineer at Orbitz
- Continuous Delivery Advocate
- DevOps into Personal life
- 5 person scrum team
- CD of Children
- theFlatironLife.com
- twitter.com/JacobTomaw
Jacob Tomaw
About Orbitz
- Founded in 2000
- Built in Chicago
- 180+ Applications
- ~5000 VMs
- 35-ish Teams
- A Great Place to Work
- Hiring!
Deployment History
-
2000: One-App, On-Demand
-
Mid 2001: One-App, Weekly
-
Spring 2002: Two-App, Weekly
-
Fall 2002: More Sites, Bi-Weekly
-
2005: Bi-weekly - technically
-
2006: Releases nearly halted
-
2006: Global Platform Refactoring
-
2010: Weekly, but...
-
2011: Bi-Weekly
-
Everything was under control
Why Go Faster?
Business needs and wants us to take measured risk.
Technically comfortable with less than the best.
When everything is under control, you are not taking enough risk.
Transformed into a learning organization.
Innovate Sooner
- Rolling Forward
- Rolling Back
- Cancelling the Release
What about obstacles?
Obstacles: A Sampling
10+ Teams - 100+ Contributors
Multiple Products
Lengthy Regression Testing
Compliance
Deployment Procedures
User Acceptance Testing
Third Party Integration
Evolving Infrastructure
Different Team Priorities
Large Change Sets
Patches
- Someone Discovers an issue
- Product Assesses Priority
- An Engineer Assesses LoE
- Product determines relative value
- Engineering fixes
- Product champions deployment
Practice
Team Effort
Pre-Prod
Prod
- A Tester Discovers an issue
- An Engineer Fixes the issue
- A new version of deployed
Rubrics
Roll Procedures
Can this remain in this environment?
Can this go to the next environment?
Existing Standards
Simple Deployment Procedures
Low Impact Deployment
Forwards or Back
A deployment is a deployment
Is the solution known?
Is the cost of the issue worth more than a deployment?
Roll Time
Roll Cost
Roll forward/Roll back variation
Certification Requirements
Product Owners
Persistent Production Problem
- Application Rolls into Production
- Several Hours Later Starts Failing
- Instincts Kick In, We try for days to save the release
- Finally Roll Back
- Problem continues
- Lost days of feedback
What we learned
Dev<-->Ops communication is not perfect
Large Change Sets are bad
Roll Back Early
Be Ready to Re-Roll
Use the Rubric
Hidden Change Sets are worse
Late Bug
- Historically Coupled Applications
- Bug Found in one of three applications
- Old Habits Kick in
- Pressure
- Hope
- Negotiation
- Pleading
- Relent
Use the Rubric
Shoulda killed The Release
Architecture Has Improved
There is always another release
Have since further decoupled the releases
Applications and Process are now Independent
"Hardest" application has achieved the most
With Great Power...
- Inversion of Accountability
- Critical Issue Found
- Poorly Responsive Team
- Cancelled the Release
The System Worked!
Used the Rubric
Don't Let Perfect be the Enemy of the Good
User Acceptance Testing
Asymmetric Testing Requirements
Two Testing Environments with Limited Scope
Historic Procedure was too costly
All Your Base Are Belong to You
Internal Cloud
Simplified Standard Deploys
Clearer Ownership
Infrastructure as a Service
On Demand Environments
Work in Progress
Never Done
This is some of what we have done.
There are other problems we have not solved yet.
There are even more problems we don't know we need to solve.
You have your own things.
Love the Journey
Everything changes
People
Market
Process
You
What Is Next For Orbitz?
- Faster
- Smaller
- Empowered Autonomy
Faster...Faster!
Fight for daily releases.
But don't fight for every day's release.
The ability and willingness are most important.
DevOpsTestProduct Teams
There is no Pre-Prod
Continuous Delivery Means Tearing Down more Silos
Empower Testers
Enable Product
Allow Specialists to focus
Build Rubrics.
What is next for you?
Make more frequent release a normal part of work.
Plan for failure.
Have only Prod Environment.
Roll forward, Roll back, Cancel the Release
By Jacob Tomaw
