Route and Batch Inference with Eight GPUs

Discuss an inference API design, concentrating on load balancing and batching. Then address a cluster with eight GPUs: one large model needs all eight GPUs, while a small model needs one GPU. The original design document is not supplied. Explain the decisions and failure modes you would evaluate; do not assume particular defects in an unseen document. Model type, traffic mix, latency targets, and whether GPU resources can be shared must be clarified. ### Part 1 — Route inference requests Explain how the service chooses a model replica and how it accounts for unequal request costs and replica readiness. #### What This Part Should Cover - Routing by model and version before balancing compatible replicas. - Work and memory estimates beyond a raw count of connections or requests. - Admission control and the handling of unavailable or overloaded replicas. ### Part 2 — Form batches Explain which requests can share a batch, how long they may wait, and how batching changes the latency-throughput tradeoff. #### What This Part Should Cover - Model, input, and execution compatibility. - Limits on batch work, memory, and queueing time. - Whether the workload needs stateless batching or state-aware generation scheduling. ### Part 3 — Allocate eight GPUs Explain how to serve the large and small models under the stated GPU requirements. Distinguish one model replica spread across GPUs from several independent replicas. #### What This Part Should Cover - Coordinated placement of the large model and the capacity left for the small model. - The consequences of exclusive allocation, model switching, and any proposed sharing. - What must change if both models require concurrent low-latency service and the current capacity cannot provide it. ### What a Strong Answer Covers - Consistent resource accounting across the router, scheduler, and model workers. - Bounded waiting and a clear overload outcome. - Measurements that could validate the proposed routing, batching, and placement choices. ### Follow-up Questions - What happens to the large-model replica when one GPU in its eight-GPU group fails? - How would you prevent continuous small-model traffic from indefinitely delaying work that needs all eight GPUs?