# Review and Scale an Inference Service Design Review a proposed inference service that places a load balancer and dynamic batching layer before model servers. Each batch must finish within 100 ms, and the service must sustain 10,000 requests per second. Identify design flaws, then explain batching, server-count estimation, overload control, and scheduling across different models. ### Constraints & Assumptions - The 100 ms statement must be clarified as batch compute time or end-to-end latency. - Request shapes and model service times may differ. - Capacity estimates must expose assumptions rather than fabricate benchmark values. ### Clarifying Questions to Ask - What are the arrival distribution, input-size mix, and latency percentile targets? - Which models share hardware, and can they be co-batched? - What behavior is acceptable when admitted demand exceeds safe capacity? ```hint Budget the latency Allocate queueing, batching, execution, and network time before selecting a batching window. ``` ### What a Strong Answer Covers - Specific weaknesses in the proposed request path. - A batcher keyed by model and compatible request shape, with bounded wait and size. - Capacity formulas using measured throughput and headroom. - Admission control, backpressure, fairness, model placement, and observability. ### Follow-up Questions 1. How would continuous batching change the design? 2. How would you prevent one slow model from consuming all accelerators?