System Design: GPU-Backed Inference Platform and API

You are designing a production inference platform to serve deep learning models (vision, ranking, and LLMs) at scale. Traffic is bursty and multi-tenant. The GPU fleet is heterogeneous (e.g., T4, A10, A100/H100), and you must support multiple models per GPU while meeting latency SLOs.

Design the system and explain key trade-offs. Specifically cover:

1. API shape
   • Synchronous vs asynchronous semantics
   • Streaming responses
   • Backwards-compatible schema and API versioning
2. Request schema and versioning
   • Request/response schemas, idempotency, and model version pinning vs aliases
3. Request routing
   • Regional routing, capacity-aware routing, and per-model queues
4. Batching strategy to maximize GPU utilization
   • Micro/continuous batching, padding, admission control; handling LLM prefill vs decode
5. Model loading and cold-start mitigation
   • Model registry, artifact formats, weight caching/warm pools
6. Multi-model hosting on shared GPUs
   • Memory management, MIG/MPS, LoRA/adapters, isolation vs utilization trade-offs
7. Autoscaling across heterogeneous nodes
   • Metrics to scale on, predictive vs reactive, bin-packing/placement
8. Placement strategy
   • Scheduling by GPU type, memory, model affinity, anti-affinity for HA
9. Observability
   • Latency (end-to-end and queueing), throughput, GPU metrics (utilization, memory, SM occupancy)
10. Rate limiting
    • Per-tenant quotas, concurrency limits, fairness
11. Multi-tenant isolation
    • Noisy-neighbor controls, security, priority tiers
12. Failure handling
    • Retries, timeouts, idempotency, circuit breakers
    • Canary and A/B rollouts, shadowing
13. High-level architecture
    • Control plane vs data plane, main components and request flows

Assume typical production SLOs (e.g., p95 latency < 300 ms for small models; streaming for LLMs), payloads up to ~10 MB, and that you need to operate multi-region.

Provide a concise, well-structured design with diagrams described in words and call out the most important trade-offs.