Review a Batched GPU Inference API Design Doc: Flaws, Throughput, Mixed-Model GPUs

This system design round is a **design-document review** rather than a whiteboard design: you are handed a short design doc for a model inference API and asked to find its weaknesses, reason about its throughput, and then extend it. No diagram drawing is expected; you talk through the document. The real document was not reported. The summary below is an illustrative stand-in built around the components the round focused on: a load balancer, a batching layer, a response cache and GPU workers. ```text Design doc (summary): Synchronous inference API Goal: serve model predictions over HTTP at a peak of 10,000 requests per second (QPS). 1. Clients call POST /predict. A layer-4 load balancer spreads connections round-robin across a fleet of stateless API servers. 2. Each API server keeps its own in-memory request queue. When the queue holds B requests (B = the model's maximum batch size), the server sends the batch to a GPU worker. A batched forward pass takes 100 ms. 3. A response cache, keyed by the exact request payload, sits in front of the API servers to save GPU work. 4. On timeout, clients retry the same request. Requests carry no identifier. 5. There are no per-client limits; every request is accepted and queued. 6. Capacity: each GPU completes B requests per 100 ms, so the fleet is sized at exactly 10,000 / (10 x B) GPUs. ``` ### Clarifying Questions - Is this a realtime, user-facing API with a latency target, or can some traffic be processed offline? What is the latency target? - How variable are request sizes (for example, input lengths), and does the 100 ms per batch depend on them? - Are responses deterministic for a given input, or sampled? - Is traffic spread evenly over time and across clients, or bursty and dominated by a few tenants? - Do all API servers and GPUs serve one model, or several? ### Part 1 — Find the flaws Walk through the document and identify its most important flaws. For each one, explain the failure it causes under realistic traffic and propose a fix. Rank them: which two or three would you insist on before this ships? ```hint Stress each section with a bad day For every numbered item, imagine low traffic, a traffic spike and a partial outage in turn, and ask what that item does to latency, cost or correctness. ``` #### What This Part Should Cover - Batching policy problems at both low and high load - How the load-balancing and per-server queues interact with batching efficiency - Whether the response cache earns its place, argued from the traffic it would actually see - Retries without request identifiers, the absence of admission control and load shedding, and the missing rate limits ### Part 2 — Theoretical versus achievable throughput Using 100 ms per batch and a peak of 10,000 QPS, derive the number of GPUs the document's own formula requires as a function of `B`. Then explain why the throughput actually achieved will be lower than this theoretical maximum, how you would size the fleet instead, and why the answer differs so much between realtime and offline inference. ```hint Question the formula's hidden assumptions The formula is only right if every batch is full and no GPU is ever idle or waiting; ask what realtime traffic and a latency target do to each of those assumptions. ``` #### What This Part Should Cover - The theoretical calculation, done correctly - The concrete sources of lost throughput in a realtime setting - The latency and utilization trade-off, and a sizing method with headroom - Why offline batch inference can approach the theoretical number ### Part 3 — Follow-up: two models on one 8-GPU pool The same pool of 8 GPUs must now serve two models with the same latency target. A batch of the large model needs all 8 GPUs at once; a batch of the small model needs 1 GPU. How do you schedule the pool? Evaluate this proposal: keep two queues, dispatch between them in proportion to each model's QPS, and prefer the large model when the two QPS values are equal. ```hint The large model needs everything at once Think about what must be true of all 8 GPUs at the moment a large batch starts, and what the small model's traffic is doing to that condition. ``` #### What This Part Should Cover - The resource cost of each model in GPU time, and the capacity condition for both to meet the target - Gang scheduling of the large model and the risk that it starves behind small batches - A concrete scheduling policy and how it protects both models' latency - Whether proportional-to-QPS dispatch and the tie-break are sound ### What a Strong Answer Covers - Critique grounded in numbers and in specific traffic scenarios, not a generic checklist - Clear prioritization of the flaws, with fixes that fit the design's constraints - Correct throughput arithmetic with explicit assumptions - A scheduling answer that reasons about GPU-time cost, fragmentation and fairness ### Follow-up Questions - The model is a large language model generating tokens one at a time. How does batching change? - How would you run a canary for a new model version on this fleet? - A single tenant sends 60 percent of the traffic. How do you protect everyone else? - Which three metrics would you put on the dashboard first, and what alert thresholds would you set?