Design scalable, highly available GenAI serving

| > Design Decision | > What We Gained (Benefit) | > What We Sacrificed (Cost / Trade-off) | > Mitigation Strategy | | --- | --- | --- | --- | | > > Continuous Batching vs. Static Batching | Up to > $10\times$ higher overall throughput and significantly lower p95 latency for short sequences. | > Higher CPU scheduling overhead and complex asynchronous memory management. | > Use high-performance C++/CUDA schedulers (vLLM/TensorRT-LLM) bound to dedicated CPU cores. | | > > Cache-Aware Routing vs. Round-Robin | > > 60–80% reduction in prefill compute , drastically reducing TTFT for shared system prompts. | > Risk of hot-spotting specific worker pods if one system prompt becomes wildly popular. | > Enforce bounded-load consistent hashing; spill overflow traffic to secondary pools if node queue > $> 50\text{ ms}$ . | | > > Multi-LoRA Serving vs. Dedicated Endpoints | > Ability to serve > hundreds of custom fine-tuned models from a single GPU pool, saving millions in hardware costs. | > Slight latency penalty ( > $\approx 10\text{ ms}$ ) when swapping LoRA weights from host RAM into GPU HBM for infrequent tenants. | > Cache top-10 most active LoRA adapters permanently in GPU High-Bandwidth Memory (HBM). | | > > Speculative Decoding vs. Standard Decode | > > > $1.5\times - 2\times$ > faster token generation speed (TPOT) without loss of quality. | > Consumes additional VRAM to host the draft model and increases complexity in the inference engine. | > Dynamically disable speculative de