## Design an LLM Inference Serving System Design a production system that serves inference requests for one or more large language models. Cover the request path from admission through token generation and explain how the system uses accelerators efficiently without allowing one long request to block all short requests. ### Constraints & Assumptions - Requests may have different prompt lengths, output limits, and model versions. - Model weights are too large to load separately for every request. - Accelerator memory must hold weights, temporary activations, and key-value cache state. - Streaming output may be required, and clients can disconnect mid-generation. - Do not assume a latency target, throughput target, model size, or accelerator count; identify the measurements needed to size the system. ### Part 1 — Define the Request and Model Lifecycle Describe the external API, validation, model-version selection, tokenization, admission control, and streaming response behavior. #### What This Part Should Cover - Stable request identity and generation parameters. - Input and output limits that protect memory and queue capacity. - Immutable model-version routing and safe rollout or rollback. - Cancellation propagation when a client disconnects. ```hint Bound work before enqueueing it Prompt tokens, requested output tokens, and model choice determine whether a request can fit before generation begins. ``` ### Part 2 — Schedule Work on Accelerators Design the worker and scheduling layer. Explain batching, key-value cache allocation, model parallelism when one model does not fit on one device, and fairness between long and short requests. #### What This Part Should Cover - Separation of prompt processing from iterative token decoding. - Dynamic or continuous batching as sequences enter and finish. - Cache-page allocation, reclamation, and out-of-memory prevention. - Tensor or pipeline parallelism only when required by the model and hardware. ```hint Batch changes over time Generation lasts many decoding steps, so a useful scheduler can replace completed sequences instead of waiting for the original batch to finish together. ``` ### Part 3 — Scale, Recover, and Observe the Service Explain replica placement, overload behavior, worker failure, deployment, and the signals used to distinguish queueing, model execution, and streaming delays. #### What This Part Should Cover - Routing to warm replicas of the requested model version. - Backpressure or rejection before accelerator memory is exhausted. - Retry boundaries that avoid silently duplicating a streamed generation. - Queue, prefill, decode, cache, token-throughput, error, and cancellation metrics. ```hint Measure each latency segment One end-to-end number cannot reveal whether time was spent waiting for admission, processing the prompt, decoding tokens, or sending them to the client. ``` ### What a Strong Answer Covers - Connects API limits and admission decisions to accelerator memory and scheduling. - Explains why continuous batching improves utilization and what fairness controls it needs. - Treats model versions, cancellation, overload, and partial streaming as explicit states. - Sizes and evaluates the design from measured workload and model behavior rather than invented capacity numbers. ### Follow-up Questions 1. How would prefix caching change memory use and isolation between tenants? 2. When would separate prefill and decode worker pools help? 3. How would you roll out a quantized model while comparing quality and latency with the current version? 4. Which request should the scheduler admit when free cache space is fragmented?