System Design: Cloud AI Inference Platform for Real-Time and Batch

Context

Design a multi-tenant cloud platform that serves machine learning models for both real-time (online) and batch (offline) workloads. The platform should support multiple model frameworks and versions, meet latency and throughput targets, and provide strong isolation, observability, and cost control.

Requirements

1. Model Packaging and Versioning
   • How will models (artifacts + code + environment) be packaged, versioned, and promoted across environments?
2. Hardware Selection
   • When to use CPU vs GPU for different models/workloads? Consider quantization/compilation and GPU sharing.
3. Autoscaling Strategy
   • Horizontal/vertical autoscaling for online and batch; scale-to-zero for idle models.
4. Request Routing
   • Routing by model/version/tenant; consistent hashing; priority and rate limiting; streaming for tokens.
5. Multi-Tenant Isolation
   • Compute/storage/network isolation; quotas; fairness; GPU partitioning.
6. SLO Targets
   • Propose latency and throughput targets for representative model types (e.g., small classifiers, embeddings, LLM text generation) and batch SLAs.
7. Cost Controls
   • Right-sizing; spot/preemptible instances; model optimization (quantization, distillation); bin packing; budgets/chargeback.
8. Observability
   • Tracing, metrics, logs; SLI/SLOs; GPU metrics; model quality/drift.
9. Safe Rollouts
   • Canary and shadow deployments; rollback criteria; guardrails.
10. Failure Handling
    • Retries/backoff; circuit breaking; regional failover; GPU OOM; degraded modes.
11. Kubernetes Integration
    • Ingress, scheduling (GPUs/MIG), operators/CRDs, HPA/KEDA, service mesh, persistent caches.
12. Model Registry Integration
    • Registry choices, signatures/schemas, lineage; CI/CD from registry to serving.
13. Industry Comparison
    • Briefly compare to managed offerings and open-source stacks.