# Design a Dynamically Batched Inference API Design a high-concurrency HTTP API for language-model inference. Clients submit individual requests, but GPU workers should process compatible requests in dynamic batches for efficiency. The service has a limited GPU pool and must balance throughput with an interactive latency target in the sub-second range. Describe admission control, request routing, batch formation, scheduling, cancellation, streaming output, GPU memory management, fairness, and overload behavior. Explain which requests are compatible in one batch and how you would measure whether batching actually improves the system. ### Constraints & Assumptions - Prompt and generated lengths vary and are not fully known at admission time. - Requests can have different models, versions, decoding parameters, and priority classes. - A disconnected client should not consume unlimited generation work. - GPU memory, not only compute, can limit batch size. - The exact latency target and traffic shape must be measured rather than invented. ### Clarifying Questions to Ask - Is the API streaming, non-streaming, or both? - Which model variants and decoding settings must coexist? - Are requests preemptible, and are there tenant quotas or priorities? - What latency percentile and throughput objective defines success? - Can prompts share a prefix cache, and what isolation rules apply? ### What a Strong Answer Covers - Queueing and batch policies tied to measurable objectives. - Memory-aware scheduling and continuous batching. - Backpressure, fairness, cancellation, and failure isolation. - Version routing, observability, and safe degradation under overload. ### Follow-up Questions 1. How would speculative decoding change scheduling? 2. How would you prevent one long generation from delaying short requests? 3. How would you roll out a new model without fragmenting batches excessively? 4. When is a larger batch slower for users despite higher GPU throughput?