Server Memory

Debugging Adventures

First Contact

First Contact

First Contact

~6:20 PM?

What requests were happening at 6:20 last night?

What was deployed? What was changed?

Investigation

Investigation

🤔

Investigation

Investigation

6:10 - 6:40 PM

Investigation

🤔

🤔

🤔

Investigation

Debugging (Puma)

memory increasing forever

😰

Debugging (Unicorn)

🔪

🔪

🔪

😰

Findings

  • /reports_dashboard/cycles was allocating >900 MB
  • Unicorn was killing workers after 60 secs

this is why we haven't seen memory problems before

UWK: per worker,  PWK: per cluster

  • With this testing, Unicorn used ~5% more RSS
  • MAX_MEMORY_WEB is not set on staging
  • MAX_MEMORY_WEB is interpreted differently for PumaWorkerKiller and UnicornWorkerKiller

Actions

  • New Configuration for MAX_CLUSTER_MEMORY
  • Add Rack::Timeout for handing request timeout
  • Incremental Rollout of Puma

Learnings

  • See Something, Say Something
  • Look at the Metric & Logs
  • Know & Build Debug Tools
  • Test Your Hypotheses
  • Pairing: Fun and Effective

💜

deck

By Tony Ta

Made with Slides.com

deck

  • 99

More from Tony Ta