Design a high-concurrency LLM inference service

You are designing an LLM inference platform that serves interactive user requests (chat/completions) on GPUs. ## Goals - Support **high concurrency** with predictable **tail latency** (p95/p99) while maintaining good **throughput**. - Optimize **GPU utilization** under real constraints: limited GPU memory, compute saturation, multi-tenant workloads. - Support **streaming** token output (server-sent events / websockets) and **non-streaming** responses. - Support **multiple models and multiple versions** of the same model (A/B, canary, rollback). - Handle **cold start** and **hot model** lifecycle management. - Be **cost-aware** (e.g., $/token) and able to trade off latency vs cost. ## Must-discuss topics 1. Sketch the **end-to-end inference pipeline**, explicitly separating **prefill** vs **decode** phases. 2. Explain what the **KV cache** is, what problem it solves, and its impact on memory/latency. 3. Batching strategy: - **Static batching** vs **dynamic batching** vs **micro-batching**. - What goes wrong when batch size is too large vs too small. 4. Scheduling under mixed request sizes: - Long context vs short context; how that affects latency and GPU memory. - How to prevent **tail latency** explosions and head-of-line blocking. 5. Request management: - When/how to **split and merge** requests (e.g., chunking long prompts, speculative approaches). 6. Multi-model/version routing: - How requests get routed to the right model/version. - Rollout/rollback and warmup considerations. ## Deliverables - A proposed architecture (components and responsibilities). - Key algorithms/policies for batching and scheduling. - Observability: the metrics you’d track and how you’d debug performance regressions. - Clear tradeoffs and failure/overload behavior.