Anthropic API Rate Limiting and Usage Quotas

What's being tested

Interviewers are probing your ability to design, implement, and operate robust API rate limiting and quota systems as part of a backend service. They want to see correct algorithm choice (token bucket/Leaky bucket/sliding window), distributed enforcement tradeoffs, client-server retry behaviour, observability, and how you reason about fairness and burstiness under production constraints. At Anthropic, this matters for protecting model-serving capacity, enforcing per-customer billing quotas, and preventing noisy neighbors from degrading latency for others.

Core knowledge

Token bucket: tokens refill at rate r (tokens/sec); bucket capacity B allows bursts; request consumes tokens; tokens = min(B, tokens + r*Δt). Use for bursty, rate-limited APIs where sustained rate r is required.
Leaky bucket vs token bucket: leaky bucket enforces constant output rate (smoothing), token bucket permits bursts then enforces average rate; pick token bucket for burst tolerance.
Fixed-window vs sliding-window vs sliding-log: fixed-window is cheap but has boundary spikes; sliding-window percentile reduces spikes; sliding-log stores timestamps (accurate but heavy) or use time-window with approximate counters (e.g., two-window weighted).
Distributed enforcement: centralized store (Redis) with atomic ops/Lua for counters or token counters offers consistency; local token buckets + periodic sync reduce latency but risk brief overcommit. Design around acceptable overage window.
Strong vs eventual consistency: atomic INCR+TTL is simple but becomes a single point; sharded counters or CRDTs reduce contention at the cost of complexity and temporary over-allowance.
HTTP semantics: return 429 Too Many Requests, include Retry-After header and human-readable quota/usage headers; provide per-key and global headers for client-side backoff.
Retry/backoff: use exponential backoff with full jitter: base * 2^attempt (capped), then sleep = uniform(0, cap). Prevents thundering-herd coordination and reduces retry storms.
Idempotency: for non-idempotent requests, require client-supplied idempotency keys (Stripe pattern) to safely retry when quotas or transient failures happen.
Metrics and SLOs: instrument allowed, throttled, rejected, quota_exhausted counters and p50/p90/p99 latency; export to Prometheus and dashboards in Grafana. Track per-tenant throttling rate for billing/alerts.
Quota models: per-second RPS, concurrent-inference slots, daily token budgets (e.g., tokens consumed by model calls). Convert diverse resources into a common cost metric where possible.
Fairness & priority: implement weighted fairness or priority queues; avoid simple first-come first-served for multi-tenant fairness. Consider leaky-bucket per-tenant + global allocator.
Throttling policies: soft limits with warning headers vs hard enforcement; consider grace periods for new customers and immediate hard limits for abuse.
Protection patterns: circuit breakers, global caps, and slow-start ramps when a key first becomes active to guard cold-start spikes.

Worked example

Design a distributed per-customer rate limiter that supports bursts and a global capacity cap. First 30s framing: ask whether the limit is per-second or longer, whether bursts are allowed (size B), and whether overages must be strictly prevented or tolerable for short windows. Skeleton answer pillars: (1) choose token bucket per-customer with refill rate r and capacity B, (2) enforce centrally using Redis with Lua script for atomic token consumption and global counters, (3) add local caching (small local token-bucket) to reduce Redis calls and accept small overcommit with reconciliation. Tradeoff to flag: central Redis gives strong correctness but is a throughput bottleneck; local buckets improve latency but can transiently exceed global capacity—acceptable if overcommit window < few seconds and your billing/monitoring adjusts. For retries, specify exponential backoff with full jitter and return 429 + Retry-After. Close by saying: "If I had more time, I'd add borrow/stealing logic for idle tenants, quota-usage telemetry per customer, and automated alerts when global capacity approaches 80%."

A second angle

Consider enforcing both per-minute RPS and a monthly token quota (different resources). Frame it as a multi-dimensional quota problem: each request consumes a small instantaneous resource (RPS slot) and a long-term budget (tokens). Enforcement pillars: (1) check instantaneous token bucket/fixed-window counters for RPS and reject immediately if exceeded; (2) atomically decrement monthly token balance (stored in Redis or backed by a transactional store) and return quota_exhausted if depleted; (3) for scale, use a write-behind pattern where short-lived requests decrement a fast cache and reconcile with durable store asynchronously to avoid latency spikes. Tradeoffs: atomic cross-checks across dimensions raise latency; you may accept eventual reconciliation for the monthly quota with strict RPS enforcement to protect system.

Common pitfalls

Pitfall: Thinking a single algorithm fits all use cases. Token bucket, leaky bucket, fixed/window each target different workload shapes; choosing the wrong one leads to surprising bursts or excessive rejections.

Pitfall: Over-relying on local caches without bounding overcommit. A tempting optimization is full local enforcement to avoid Redis calls; without a reconciliation window this can exceed global capacity and break fairness.

Pitfall: Ignoring client-side behaviour and observability. Returning bare 429 without Retry-After, usage headers, or clear billing signals leads to poor client UX and repeated retries that amplify load; always provide actionable headers and instrument retry rates.

Connections

Interviewers may pivot to adjacent topics like API gateway scaling and placement (Envoy, Nginx), distributed coordination techniques for counters (consistent hashing, sharding), or billing/usage pipelines that consume quota telemetry. They might also ask about model-serving capacity planning and how rate limiting interacts with autoscaling.