F*ckUp Gracefully

Radim Klaška

Morpht.com & Drupal.cz

Who are you?

"Quick" deploy

Background

• Site rescue project
• Big site (2+ mil. nodes)
• Redmine & Jira
• Few processes to follow
• External QA team

Task

• Deploy the whole sprint
  • updates
  • new modules
  • optimization to search queries
  • search index changes
  • theme changes

First clue: this random message

Deploy time

Solution

• d.o/project/node_access_rebuild_progressive
• Search API: Mark for reindexing

Damage report

• Server load
• Developer time
• Client trust probably

Retrospective

• Cloudflare cache saved us
• Company's first incident report!
• Communication problems (source of truth)
• Client was informed about possible downtime

Monthly Membership is not

a subscription

Background

• D7, Ubercart
• Site selling access to files
• Issue repored by client

Report

Solution

Damage report

• Developer time
• Client's time
• ~2 customers of our client affected

IPv6 just in time

+

we need to log everything

Background

• Site logging every "view" event
• Big DB (just one table)
• Czech ISP switching to IPv6

Nope.

All good.

We want that data.

While working on that...

Damage report

• None?
• Developer time was paid
• We actually improved the site

Retrospective

• Hey, we did pretty good! :)
• Enjoy the little things

Keep your cool,

but iterate!