GPU Inference API Serving

What's being tested

Interviewers test your ability to design a multi-tenant GPU-backed inference service that meets explicit latency SLOs while maximizing accelerator utilization and operational reliability. Expect to demonstrate end-to-end thinking: API contract and lifecycle, scheduling/admission control, batching and accelerator-level optimizations, model/version management, and observability for p99 latency and cost. Anthropic cares about pragmatic tradeoffs (latency vs throughput, isolation vs utilization) and clean, testable designs a Software Engineer can implement and operate.

Core knowledge

• API modes: synchronous streaming vs asynchronous / batch. Synchronous streaming (text-generation) needs low tail latency and chunked responses; asynchronous accepts job submissions and returns a job id for later polling or callback.

• SLO/SLI design: define p50, p95, p99 latency SLOs, throughput SLO, and error-rate; instrument request_latency, queue_time, inference_time, gpu_utilization. Use these to drive admission control and autoscaling.

• Queueing & admission control: model arrivals with $λ$, service rate $μ$ per GPU; utilization $ρ=λ/μ$. Use Erlang-C or M/M/c approximations to estimate queue wait $W$ and dimension capacity for a target p99.

• Batching math: throughput ≈ batch_size / batch_service_time; batch_service_time grows sublinearly in batch_size until memory/compute saturation. Optimize batch size to hit a target latency budget while maximizing throughput.

• GPU runtime constraints: CUDA context creation is expensive; memory fragmentation, model size, and kernel launch overheads limit effective concurrency. Use MIG, CUDA MPS, or container-level isolation for multi-tenancy tradeoffs.

• Model placement & sharding: small models fit on one GPU; large LLMs require tensor/model parallelism or pipeline parallelism across multiple GPUs, increasing inter-GPU communication (NVLink, NCCL) and latency.

• Dynamic batching & sequence batching: for autoregressive workloads, dynamic batching must respect token-level latency; implement sequence-aware batching that groups similar-length sequences and supports early flushing when latency budget is reached.

• Memory & input constraints: model + activation footprint must fit GPU memory; use quantization (int8, fp16) and offloading (CPU/RAM or NVMe) carefully: quantization reduces memory and latency but can alter quality.

• Scheduler & placement policies: implement a centralized scheduler (or per-cluster agent) that considers GPU memory, compute load, model residency, queue lengths, and tenant isolation; support pre-warmed models to avoid cold starts.

• Autoscaling & warm pools: scale at two levels: stateless worker replicas (if using model server) and GPU cluster capacity (node autoscaler). Maintain a warm-pool of preloaded models to avoid cold-starts for p99-sensitive paths.

• Failure modes & retries: detect OOM, driver resets, and preemption; isolate failures per request, use idempotency keys, and circuit-breakers to avoid retry storms that harm GPUs.

• Observability & debugging: correlate traces across API gateway -> scheduler -> model server -> GPU for a single request. Export gpu_memory, gpu_utilization, queue_depth, and batch_size_histogram for root-cause of p99 spikes.

• Cost & accounting: attribute GPU time per tenant/model using wall-clock and batch accounting. Consider spot/preemptible GPUs for non-SLO workloads and on-demand for SLO-critical paths.

Worked example — Design a GPU inference API

First 30 seconds: clarify SLOs (target p99 latency), supported API patterns (streaming vs non-streaming), concurrency characteristics, model sizes (single-GPU or multi-GPU), multi-tenancy/isolation requirements, and cost constraints. Skeleton pillars: (1) API contract & lifecycle (endpoints for POST /v1/generate synchronous, POST /v1/jobs asynchronous with job_id); (2) Request router & admission control that accepts/queues requests, enforces per-tenant rate limits, and computes batch eligibility; (3) Scheduler & runtime that places models into GPU resident pools, performs dynamic batching, and invokes Triton-style model servers; (4) Observability & autoscaling driving warm-pools and node scaling. A specific tradeoff: prioritize p99 by capping batch size and pre-warming models at the cost of lower throughput and higher GPU idle time; alternatively maximize utilization with larger batches but accept higher p99. Close: propose short experiments and metrics (simulate workloads, measure p99 vs throughput) and say "if more time, I'd design the exact admission-control policy, simulate it with traces, and prototype dynamic batching thresholds."

A second angle — Design a batch inference API

Batch inference is primarily asynchronous and throughput-first; design focuses on job lifecycle, chunking, checkpointing, and idempotent retries. Key changes: expose POST /v1/batches returning job_id, support resumable checkpoints for large datasets, and provide progress and partial-result streaming. Worker pool design uses a job queue with parallel workers that can aggregate multiple small inputs into GPU batches internally, but scheduling is simplified because strict p99 latency is relaxed. Emphasize durable storage for inputs/outputs (S3), rate-limits to avoid runaway costs, and cost-aware scheduling (spot instances for low-priority batches, guaranteed capacity for high-priority jobs). Also plan for per-job SLAs and shardability of large datasets to parallelize across GPUs.

Common pitfalls

Pitfall: Designing for average latency or throughput only. Many candidates tune batch sizes to maximize throughput and ignore queueing effects on p99, causing SLO misses under bursty traffic. Always dimension by tail latency and simulate with realistic variance.

Pitfall: Treating GPUs like infinitely shareable CPUs. Ignoring CUDA context costs, memory fragmentation, or multi-tenant interference leads to noisy neighbors and unpredictable p99 spikes. Explicitly model GPU residency and use hardware isolation primitives.

Pitfall: Over-architecting without measurable signals. A tempting deep design adds complex model-parallel pipelines; better answers first propose measurable experiments (traces, A/B of batching strategies) and incremental rollout plans with observability hooks.

Connections

Interviewers often pivot to adjacent topics: model rollout & canary deployments, feature-store/real-time retrieval for contextualized inference, or scheduling/resource management at cluster scale. Be ready to discuss how the inference design ties into CI/CD for models, monitoring regression in generation quality, and cost allocation per tenant.

Further reading

• NVIDIA Triton Inference Server docs — practical model-serving patterns and batching/runtime features.

• Ray Serve docs — patterns for scalable deployment and request routing for Python-first model servers.

• Google Borg paper — cluster scheduling principles useful for large-scale GPU placement and multi-tenant policies.