LLM API Gateway, Rate Limits, And Abuse Prevention

What's being tested

Designing a robust API gateway that enforces scalable rate limits and defends against abuse requires balancing correctness, latency, and operational complexity. Interviewers probe your ability to pick and justify distributed enforcement strategies (consistency vs. availability), quantify capacity and burst behavior, and integrate detection/mitigation without blowing up latency or false positives. Expect to explain failure modes, instrumentation, and one or two concrete implementation sketches that could be built in a week.

Core knowledge

Rate-limiting primitives: understand fixed-window, sliding-window log, sliding-window counter, token-bucket, and leaky-bucket; token-bucket: capacity = burst, refill rate r tokens/sec, consume k tokens per request.
Tradeoffs: accuracy vs. latency: fixed-window is cheapest (O(1)) but produces spikes; sliding-window reduces spikes at higher storage cost; token-bucket supports bursts naturally.
Distributed enforcement patterns: centralized store (Redis) with atomic ops (Lua scripts) for global counters, or local token-buckets per gateway node with periodic sync / conservative quotas for cross-node coordination.
Sharding and scale: use consistent hashing to shard counters across a Redis cluster; for approximate needs, use Count-Min Sketch to reduce memory at the cost of over-counting.
Latency budget: putting enforcement on the critical path must meet SLOs (e.g., add <5–10ms); prefer local checks for p99-sensitive paths and async accounting for billing.
Fairness and multi-tenancy: support per-API-key, per-organization, per-IP limits, and weighted fairness (e.g., deficit round-robin) to prevent noisy neighbors.
Failure modes & resilience: decide fail-open (availability) vs fail-closed (safety); use degraded-mode heuristics (e.g., lower default limits) and circuit breakers to avoid cascading failures.
Backpressure & client signaling: return 429 Too Many Requests with Retry-After, expose remaining-quota headers, and support soft-limits for graceful throttling.
Abuse detection signals: behavioral (high request rate, repeated malformed prompts), credential abuse (rapid key rotation), and content heuristics (repeated injection patterns); combine rules + lightweight statistical detectors.
Mitigation tools: automated throttling, temporary key suspension, CAPTCHA or proof-of-work for unauthenticated flows, connection blackholing for verified abusers.
Operational metrics: track p50/p95/p99 latency, throttle rate, quota exhaustion rate, unique active keys, false-positive rate for abuse detection, and SLO burn.
Clock and atomicity issues: avoid relying on client clocks; use monotonic server time and atomic operations (e.g., INCRBY with expiry) to prevent race conditions and skewed bursts.

Worked example — "Design an LLM API Gateway that enforces per-customer rate limits and prevents prompt-injection abuse"

First 30s: clarify scope — are limits per API key, per account, per IP? Is enforcement global across regions? Are latency SLOs strict (p99 < X ms)? Are we blocking for safety or simply throttling? Skeleton: (1) edge gateway (Envoy/NGINX) enforces local token-bucket for low-latency checks, (2) centralized Redis rate-limiter for cross-node single source of truth (Lua script for atomicity), (3) light-weight content heuristics filter (regex/signature) plus async ML-based signal pipeline for deeper analysis. Key tradeoff: local buckets reduce latency but can allow slightly higher aggregate bursts; centralized enforcement is exact but adds cross-AZ latency—choose local-first with periodic reconciliation if p99 latency matters. Failures: on Redis outage prefer fail-open with reduced default quotas and increased logging, or fail-closed if safety-critical. Close by saying: with more time I’d prototype the Redis Lua scripts, write canary tests to measure p99 impact, and add a shadow-mode to tune abuse-detection thresholds.

A second angle — "Preventing abuse from distributed clients rotating API keys and IP addresses"

Here the attacker evades per-key/IP limits by rotating credentials. Emphasize device- or behavior-level signals: fingerprint request headers, TLS client hello fingerprints, rate-of-new-key-creation, and anomaly scoring aggregated per account. Architecturally, push detection into a streaming analytics pipeline (lightweight enrichment at the gateway, heavy scoring off-path) and apply rapid short-term mitigations (temporary global soft-throttle or proof-of-work) while the account’s historical risk score is computed. The tradeoff is privacy and false positives: more aggressive fingerprinting improves detection but raises privacy and operational complexity.

Common pitfalls

Pitfall: using fixed-window counters by default.
Fixed-window counters are attractive for simplicity but allow 2x bursts at window boundaries; interviewers will flag this. Show you know sliding or token-bucket alternatives and quantify burst behavior.

Pitfall: not clarifying quota semantics.
If you don't ask whether limits are per-key or per-org, the interviewer will test you with multi-API-key tenants. Always state assumptions about identity granularity and how shared quotas behave.

Pitfall: ignoring degraded-mode behavior and clock skew.
Designs that assume always-available Redis or perfectly synchronized clocks break in outages; explain fail-open vs fail-closed choices and how monotonic server time + atomic ops (Lua) prevent race conditions.

Connections

Expect quick pivots to adjacent areas: authentication & authorization (how tokens/roles affect quota), observability & SLOs (how throttling counts against customer-visible SLOs), and distributed caching/consensus (how to shard and replicate counters consistently).