Design a Model Inference API and Optimize Its GPU Usage

Design an inference API: a service that lets clients send inputs to a trained machine-learning model running on GPUs and get the model's outputs back. The round covered the end-to-end design of the API and the serving system behind it, and the follow-up questions went into how to optimize the GPU side. The model and the traffic were not specified. Treat them as clarifying questions; where you need a working assumption, a large language model that generates text token by token is a natural one, because it makes both the API and the GPU questions concrete. ### Constraints and Clarifications - Not given: the model type and size, request volume, latency targets, input and output lengths, whether responses stream, and how many GPUs are available. Ask, or state your assumptions. ### Clarifying Questions - What does the model do: generate text token by token, or produce one output per input in a single forward pass (classification, embeddings)? - How large is the model, and does it fit in one GPU's memory or must it be split across several GPUs? - Which latency matters most: time to the first output token, time to the full response, or throughput for offline batch jobs? - Do clients need streamed responses, and how long are typical inputs and outputs? - Who are the clients: is there per-customer authentication, quotas or rate limits, and are there priority tiers? - What request volume and burst pattern should the system handle? ### Part 1 — The API and the serving system Design the public API and the system behind it: how a request enters, where it waits, how it reaches a GPU, and how the result gets back to the client. ```hint Where requests wait GPUs are the scarce, expensive resource. Decide where a request waits when every GPU is busy, what decides which request runs next, and what the client sees while it waits. ``` ```hint Generation is not one step If the model produces its output one token at a time, a request occupies a GPU for many steps, and different requests need very different numbers of steps. Think about what that does to the way you group requests and to how the response is returned. ``` #### What This Part Should Cover - The API surface: request and response shape, streaming, errors, authentication, and limits - The request path from the gateway through queueing and scheduling to the model servers and back - How capacity is shared among clients: rate limits, admission control, and behavior under overload - Scaling and failure handling: adding model replicas, rolling out new model versions, and a GPU worker that dies mid-request ### Part 2 — Getting more out of the GPUs The interviewer then asked how to optimize the GPU side. Explain what limits throughput and latency on a GPU for this workload, and what you would change to serve more requests per GPU without missing the latency targets. ```hint Find what the GPU waits on During one generation step for a single request, ask whether the GPU spends its time on arithmetic or on reading the model weights and stored per-request state from memory. Then ask what serving several requests in the same step changes about that. ``` #### What This Part Should Cover - What bounds the time of a GPU step for this workload, and how grouping requests changes it - GPU memory management: model weights versus per-request state, and what limits how many requests fit at once - Techniques that trade precision, latency, or hardware count for throughput, each with its cost - How to measure the effect: throughput, latency, and GPU efficiency metrics ### What a Strong Answer Covers - Settles the model type, latency targets and traffic before committing to a design - A clear API, including streaming and error semantics, and a request path that explains where requests queue and how they are scheduled onto GPUs - Overload behavior: admission control, per-client limits and fairness instead of unbounded queues - A correct account of which phases of GPU inference are limited by memory bandwidth and which by compute, and optimizations that follow from it - Failure handling, safe model rollout, and the metrics that show whether the system meets its targets ### Follow-up Questions - A few clients send very long inputs and crowd out everyone else. How do you keep latency fair? - The model no longer fits in one GPU's memory. How do you split it, and what does that cost per request? - Traffic falls sharply at night. How do you avoid paying for idle GPUs without hurting the morning ramp-up? - A new model version must be rolled out without dropping in-flight streamed responses. How?