Design a GPU inference API
Company: Anthropic
Role: Software Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Onsite
##### Question
Design a scalable, GPU-backed inference API for serving multiple deep learning models (including large autoregressive models such as LLMs) to product services. The system must support low-latency online inference with clear SLOs, scale from a small deployment to high traffic, and serve multiple model versions and tenants across a shared, heterogeneous GPU fleet. Walk through the architecture end to end and reason about bottlenecks with metrics rather than scaling every component blindly.
Discuss:
1. **Public API shape and request lifecycle.** What does the synchronous prediction endpoint look like (request/response fields, request schema across task types, idempotency, tenant identity, model version selection)? How do you version the API itself versus the models behind it (path/header API versions, aliases such as `stable`/`latest` versus exact pins)? Which protocol do you choose (gRPC vs HTTP/2 + SSE), and when do you also need an async / job-based API with polling or webhooks, plus streaming responses?
2. **Core architecture and data flow.** API gateway, frontend/CPU validation and preprocessing, scheduler/queue, dynamic batching layer, GPU inference workers, model registry/artifact store, async job store, and control plane. Describe the request flow through these components for both the sync and async paths.
3. **Independent scaling of CPU and GPU components.** Which signals drive CPU autoscaling vs GPU-pool autoscaling, and why are they decoupled?
4. **Diagnostic scenario:** what would you do if CPU utilization is low but the GPUs are saturated? Walk through how you confirm the bottleneck and the ordered set of actions you'd take.
5. **Dynamic / continuous batching and SLO-aware scheduling.** How do you form batches under latency deadlines (micro-batching for vision/classification vs continuous batching for LLMs, prefill vs decode), ensure per-tenant fairness, and apply admission control, queueing and backpressure?
6. **GPU memory management.** Weights residency, KV/paged-attention cache sizing, quantization, tensor parallelism, fragmentation, warm pools and eviction, and per-tenant isolation.
7. **Model versioning, A/B routing, shadow traffic, canarying, and rollbacks.** How does the registry and router support traffic splits, shadow evaluation, and safe rollout/rollback?
8. **Autoscaling and placement across heterogeneous GPU nodes** (different GPU types, throughput curves, bin-packing and VRAM headroom, model/GPU compatibility, affinity and anti-affinity, prewarming, spot/preemptible handling).
9. **Model loading, warmup, and cold-start mitigation**, including the artifact cache hierarchy (object store → node NVMe → pinned host RAM → VRAM), warm pools, and lazy adapter (LoRA) loading.
10. **Reliability, observability, capacity planning, cost controls, and security** (retry semantics, timeouts, circuit breakers, GPU-fault handling, graceful degradation, latency breakdown metrics, rate limiting and per-tenant quotas/billing, supply-chain and tenant isolation).
11. **Request routing across regions.** How do global and regional routing decide where a request runs (geo/latency, health, capacity), when do you make routing sticky per tenant or model to preserve warm caches, and how do you fail over or shed to an adjacent region without hot-spotting it?
12. **Multi-model hosting on shared GPUs.** When do you dedicate a GPU to one model versus colocating several? How do you share base weights across LoRA adapters, and what are the trade-offs between MIG partitioning, MPS, and plain time-slicing for isolation?
13. **Key trade-offs.** Summarize the main tensions in the design: sync vs async, batching vs tail latency, isolation vs utilization, colocation vs fragmentation, precision vs accuracy, and routing stickiness vs load balance.
Overview: This Anthropic ML system design onsite asks you to design a multi-tenant, GPU-backed inference API that serves many models and versions -- including large autoregressive LLMs -- under explicit latency SLOs. It spans API and schema design, streaming and async jobs, continuous batching with prefill/decode separation, KV-cache and GPU memory management, multi-model hosting on shared GPUs, heterogeneous autoscaling and placement, multi-region routing, canary rollout, and reliability and cost controls. The signature probe is the diagnostic scenario where CPU utilization is low but the GPUs are saturated.