Fire Fighting: A developer's story!

Ahmed Eid

Software Developer

Booking.com, Amsterdam

crouchsays

Agenda

• What to expect?
• What brought me here?
• What is a fire?
• Fire-fighting vs. The team
• Mapping out dependencies
• Feature toggles
• Abuser Stories
• The unknown unknowns!

What to expect?

• Why fires aren't only harming your SLIs?
• A pro-active approach to fire-fighting.
• What could they have done better?

What brought me here?

• Same process, different problem
• The scale matters
• Less fire-fighting = Higher productivity

What is a fire?

Anything that would require you to drop whatever you have in hands and start working on.

IOW, anything that would require an emergency re-allocation of resources!

What leads to them?

• Cutting corners while solving problems
• Too many problems and no enough time
• Allowing others control the project agenda
• No priorities, everything is urgent

Fires are harmful

• Disturb the planned work
• Hero culture
• Stressful for fighters
• Patches make it chronic

Dependencies

• Think about your subsystems and dependencies
• Document impact in case of failure
• Think of fallbacks or degradation of service

Features

• What are they?
• Release Toggles vs Long-lived Switches
• Document them:
  • When to use?
  • What is the impact?

Abuser Stories

• Evil users?!
• It's important to change hats
• At least think of it so you are ready

Facebook

As an abuser, I'd like to live stream violent content(video) in order to make it viral.

Twitter

As an abuser, I want to post social engineering-based scams and make them go viral.

The unknown unknowns

• You don't know what you don't know
• Users use systems in unexpected ways
• Log messages & metrics are your best friends

Assess
Contain
Investigate

Thanks :)