GPU Backend Serving and Scheduling for Cloud

What's being tested

This topic tests whether you can design a reliable distributed backend that schedules scarce accelerators while meeting latency, throughput, and isolation requirements. The interviewer is probing your ability to decompose the system into request routing, queueing, placement, execution, autoscaling, and observability rather than treating “GPU scheduling” as a single queue. Google cares because inefficient GPU utilization directly increases serving cost, while poor scheduling can violate `p99` latency SLOs, starve tenants, or make failures difficult to contain.

Core knowledge

• Workload characterization determines the scheduler’s objective: online inference usually optimizes admission latency and `p99`, batch inference optimizes throughput and cost, and training optimizes completion time and gang placement. Ask about model size, GPU memory, request-rate distribution, sequence lengths, deadlines, and preemption tolerance.

• A serving path commonly contains a stateless frontend, model-aware router, per-model queue, scheduler, GPU worker pool, and durable control plane. Keep request metadata and placement state separate from model weights; use systems such as `Spanner`, `etcd`, or a strongly consistent service for assignments, while data-plane workers remain horizontally scalable.

• Dynamic batching combines requests arriving within a bounded window, amortizing kernel-launch and transfer overhead. The scheduler must cap both maximum batch size and maximum batching delay; a useful constraint is queue_wait + compute_time <= latency_budget, with separate policies for interactive and throughput-oriented traffic.

• GPU placement is multidimensional: device memory, compute capacity, topology, model weights, CUDA/runtime compatibility, and locality all matter. A node with four nominally idle GPUs may be unusable for a model requiring one contiguous 40-GB device or high-bandwidth peer-to-peer links.

• Bin packing is the core allocation problem and is generally NP-hard, so production schedulers use heuristics such as first-fit decreasing, best-fit, dominant-resource fairness, or scoring functions over free memory, utilization, topology, and fragmentation. Reserve headroom for runtime allocations; “free memory” from telemetry is not always safely allocatable.

• Gang scheduling is required when a job needs multiple GPUs simultaneously, such as distributed training or tensor-parallel inference. Reserve all members before launch; otherwise, partial placement creates deadlock-like waste; use leases and rollback when a worker fails during allocation.

• Multi-tenant fairness requires explicit policy. Weighted fair queuing or DRF (Dominant Resource Fairness) can allocate proportional shares, while priority classes handle production versus best-effort traffic. Add per-tenant quotas, concurrency limits, and aging to prevent low-priority work from starving indefinitely.

• Preemption improves utilization but has costs: lost GPU cache state, checkpoint overhead, request retries, and tail-latency spikes. Prefer preempting checkpointable batch work rather than active interactive requests, and make retries idempotent with request IDs and bounded retry budgets.

• GPU execution is often bottlenecked by more than kernels. Account for host-to-device copies, model loading, kernel launch overhead, synchronization, KV-cache growth for generative models, and network transfers. Persistent workers and warm model replicas reduce cold-start latency but consume scarce memory.

• Autoscaling should use queue depth, queue age, arrival rate, service time, and GPU utilization—not utilization alone. A simple capacity estimate is $\text{required replicas} \approx \lceil \lambda \times S / C \rceil$, where $\lambda$ is arrival rate, $S$ mean service time, and $C$ effective concurrency; validate against `p95`/`p99` latency because averages hide saturation.

• Backpressure protects the system when demand exceeds accelerator capacity. Bound queues, reject or shed work with an explicit status, enforce deadlines, and avoid unbounded buffering. A queueing-theory warning is that as utilization $\rho$ approaches 1, waiting time rises sharply even when average service capacity still appears sufficient.

• Fault handling must distinguish request, worker, node, control-plane, and GPU-level failures. Use heartbeats and leases to detect dead workers, quarantine unhealthy devices, replay only safe requests, and reconcile control-plane assignments with actual worker state after a scheduler restart.

• Observability should expose queue wait, execution time, batch size, admission rejections, GPU memory, SM utilization, device errors, model-load time, and per-tenant fairness. Track `p50`, `p95`, and `p99` separately by model, region, priority, and request shape; a fleet-wide average can conceal one overloaded model or tenant.

• Isolation can use process boundaries, containers, cgroups, Kubernetes device plugins, or hardware partitioning such as NVIDIA `MIG`. `MIG` improves predictable partitioning for supported GPUs, but reduces flexibility and may leave unusable slices; time-slicing improves sharing but provides weaker latency and fault isolation.

• Placement should account for failure domains. Spread replicas across hosts, racks, zones, and power domains where availability requires it, but keep distributed jobs within a low-latency GPU topology when performance requires it. Treat locality and resilience as competing objectives, not automatic wins.

Worked example

No titled interview question was supplied, so use this representative prompt: “Design a multi-tenant GPU inference serving platform.” In the first 30 seconds, clarify whether the workload is synchronous inference or batch processing, the target `p99`, model sizes, request rate, GPU types, regional scope, tenant-isolation requirements, and whether stale or rejected requests are acceptable. State an assumption such as synchronous requests with a 200-ms `p99` target, several models, bursty traffic, and multiple GPU classes. Organize the answer around four pillars: request routing and bounded queues, model-aware batching and GPU placement, fairness/autoscaling, and failure handling/observability. Explain that a control plane tracks model versions, worker leases, quotas, and reservations, while workers keep hot model replicas and report health and capacity. Explicitly flag the tradeoff between packing models densely for utilization and reserving spare capacity for fast failover and predictable tail latency. Close by saying that with more time you would quantify capacity using production traces, test overload and node-loss scenarios, and define rollout and rollback behavior for model versions.

A second angle

A related prompt might ask how to schedule distributed training jobs across a heterogeneous GPU fleet. The same principles apply, but the dominant constraint changes from per-request latency to gang placement, checkpoint recovery, and completion time. You would emphasize topology-aware allocation, queue fairness, preemption at checkpoint boundaries, and reservations for all GPUs before launch. Unlike online inference, batching requests is irrelevant, and delayed admission may be preferable to starting a job that cannot obtain its full placement.

Common pitfalls

Pitfall: Treating GPU utilization as the only capacity metric. A device can show low average utilization while queues grow because the wrong GPU memory size, topology, or model-specific workers are saturated; measure queue age, memory pressure, and service time by workload.

Pitfall: Saying “use `Kubernetes`” without describing the scheduling policy. `Kubernetes` can provide primitives such as device plugins and deployments, but you still need model-aware placement, batching, quotas, gang semantics, leases, and overload behavior.

Pitfall: Ignoring tail latency and failure recovery. “Add more GPUs” is not a complete answer when cold model loads, stragglers, node loss, or unbounded retries create cascading overload; state explicit SLOs and a bounded degradation path.

Connections

The interviewer may pivot to distributed rate limiting, consistent hashing, queueing theory, leader election, distributed tracing, or `Kubernetes` scheduling primitives. For generative models, expect follow-ups on KV-cache placement, continuous batching, tensor parallelism, and admission control.

Further reading

• Orca: A Distributed Serving System for Transformer-Based Generative Models — introduces iteration-level scheduling and continuous batching for generative inference.

• Gandiva: Introspective Cluster Scheduling for Deep Learning — useful for GPU-aware scheduling, time slicing, and workload profiling.

• Dominant Resource Fairness — foundational paper for fair allocation across multiple resource dimensions.