PagerDuty: Production Support as Front End Developer
Xinjiang Shao
2021-7-29
Agenda
- The Mindset
- Incident Examples In the past
- Most Common Questions
- Tools
- Demo
The Mindset
- Stay Calm
- On-Call !== Bug Triage
- Prefer to no have any code changes when being on call
- Restart Service (e.g Flip Blue/Green)
- Content republish
- Site config to disable feature
- Data changes or direct the request to the right team
Incident #1
CMDM is down
Observation:
- Logged in users cannot checkout in GNTC and MRTN
- The new account cannot be created, existing user cannot log in
- Coupons cannot be clipped
Resolution:
We published a system-wide notification
Incident #2
Client Log 503/413 Status
Observation:
An increasing amount of client log requests
Resolution:
- The rate limit is added from Cloudflare for client log
- Add new site config to control the intervals of sending client log
- New ways of sending logs in batch from azure event hub
Incident #3
Missing PodBag
Observation:
Missing podbag for delivery service, however the location config indicated the podbag is enabled
Resolution:
- Ask bus-system dev to fix the data configuration
- Gracefully turn off podbag if the product info cannot be retrieved after the incident
Incident #4
iOS App Native Login
Observation:
App users cannot login
Resolution:
- Disable native login from site config
Most Common Questions for Front End Devs
- What kind of functionality would be impacted if we restart service x (e.g loyalty account API)?
- Are our users(customers) recovered from the incident?
- Could we disable feature x (e.g client log)?
- When did certain incident start?
Common Root Causes for Other Teams
- Copient/Quotient Offers
- Informix DB perf
- Too many concurrent users
- MDM
- Running out of disk
- DNS server
- Firewall outage
- Expired SSL certificates
- VMWare in QTS datacenter
Tools
- Datadog RUM
- Client Log in Splunk
- JIRACore
- Sourcemap in Prod
- SuperUser
- Webbase (BusSys)
- CloudFlare (Security Team)
- Shape Security (Security Team)
- Optimizely (Product Team)