Per-customer API rate limiting

Picking an algorithm, a key and the right response for an API you charge for

What to limit by

  • Customer (API key): key the bucket by customer, not by key string, or a customer with five keys gets five times the limit
  • IP address: for sign-up, login and failed key checks, at the edge
  • Global: for a shared upstream dependency, paired with per-customer limits

Rate limits and quotas are different tools

  • A rate limit works in seconds or minutes and recovers on its own
  • A quota covers a billing period. Going past it is a billing decision
  • Answer both with 429, but say which one it was and set Retry-After to match

Algorithms

  • Fixed window: simple, but a client can send 2x the limit across a window boundary
  • Sliding window log: exact, but stores a timestamp per request
  • Sliding window counter: two counters, approximate, usually good enough
  • Token bucket: two numbers per customer. Burst size and refill rate read naturally in a plan

Counting usage for billing

  • Count in memory and write in batches, not one database write per request
  • Give each batch an ID so a retry after a lost reply doesn't count twice
  • Flush on graceful shutdown. A hard kill loses counts that weren't written

Read more