PagerDuty: Production Support as Front End Developer

Xinjiang Shao

2021-7-29

Agenda

• The Mindset
• Incident Examples In the past
• Most Common Questions
• Tools
• Demo

The Mindset

• Stay Calm
• On-Call !== Bug Triage
• Prefer to no have any code changes when being on call
  • Restart Service (e.g Flip Blue/Green)
  • Content republish
  • Site config to disable feature
  • Data changes or direct the request to the right team

Incident #1

CMDM is down

Observation:

• Logged in users cannot checkout in GNTC and MRTN
• The new account cannot be created, existing user cannot log in
• Coupons cannot be clipped

Resolution:

We published a system-wide notification

Incident #2

Client Log 503/413 Status

Observation:

An increasing amount of client log requests

Resolution:

- The rate limit is added from Cloudflare for client log

- Add new site config to control the intervals of sending client log

- New ways of sending logs in batch from azure event hub

Incident #3

Missing PodBag

Observation:

Missing podbag for delivery service, however the location config indicated the podbag is enabled

Resolution:

- Ask bus-system dev to fix the data configuration

- Gracefully turn off podbag if the product info cannot be retrieved after the incident

Incident #4

iOS App Native Login

Observation:

App users cannot login 

Resolution:

- Disable native login from site config

Most Common Questions for Front End Devs

• What kind of functionality would be impacted if we restart service x (e.g loyalty account API)?
• Are our users(customers) recovered from the incident?
• Could we disable feature x (e.g client log)?
• When did certain incident start?

Common Root Causes for Other Teams

• Copient/Quotient Offers
• Informix DB perf
• Too many concurrent users
• MDM
• Running out of disk
• DNS server
• Firewall outage
• Expired SSL certificates
• VMWare in QTS datacenter

Tools

• Datadog RUM
• Client Log in Splunk
• JIRACore
• Sourcemap in Prod
• SuperUser

• Webbase (BusSys)
• CloudFlare (Security Team)
• Shape Security (Security Team)
• Optimizely (Product Team)

Demo Time