# Design a Low-Latency GPU Inference Service Design an online inference service backed by GPU servers. Explain how dynamic batching, request deadlines, load balancing, and optional caching affect latency and throughput. Derive theoretical maximum throughput from batch service time, then explain why sustainable online throughput can be lower when a tail-latency target must be met. ### Constraints & Assumptions - Requests arrive online and have an explicit latency deadline. - A GPU worker processes a batch at a measured batch-dependent service time. - Traffic can be bursty and request shapes may differ. - Caching is optional and must be justified by repeat rate, correctness, and invalidation cost. ### Clarifying Questions to Ask - What are the target latency percentiles and request arrival distribution? - Can requests with different shapes or model versions share a batch? - Is the model stateless and deterministic enough for a cache key to be safe? ```hint Include queueing time A request's latency budget is shared by admission, batching wait, execution, and response handling. ``` ```hint Balance on work GPU queue depth and estimated remaining work reveal more than request count alone. ``` ### What a Strong Answer Covers - Capacity arithmetic that distinguishes offline saturation throughput from deadline-constrained throughput. - A batching scheduler with maximum wait, size, and shape compatibility rules. - A load-balancing signal stronger than random or round robin when queues are uneven. - Backpressure, overload behavior, observability, and a measured cache decision. ### Follow-up Questions - How would the scheduler prevent a large batch from starving short-deadline requests? - How would you decide whether adding one GPU or reducing maximum batch wait gives the better latency improvement?