GPU Scheduling And Resource Management

What's being tested

Interviewers are probing your practical knowledge of GPU resource tradeoffs and the scheduling strategies an ML Engineer uses to reliably run training and inference workloads at scale. Expect to justify choices that balance throughput, latency, cost, and isolation for multi-tenant ML workloads, and to show you can operationalize solutions (checkpointing, preemption, monitoring) rather than redesigning kernel schedulers or datacenter networks.

Core knowledge

GPU memory (VRAM) versus model size: VRAM limits maximum model + optimizer state; use gradient accumulation or activation checkpointing when batch-size or model params exceed memory. Memory footprint ≈ params*4B (FP32) + optimizer states.
Compute vs memory-bound: identify whether a model is compute-bound (SM utilization high, limited by FLOPS) or memory-bandwidth-bound (limited by DRAM/PCIe). Use nvidia-smi and DCGM metrics to measure SM utilization and memory throughput.
Mixed precision: switching to FP16 / bfloat16 reduces memory and increases throughput via tensor cores; manage accumulation with a loss-scaling policy to avoid underflow. Typical speedups 1.5–3× depending on hardware.
Data-parallel vs model-parallel: Data-parallel (DDP/NCCL AllReduce) is simplest for most models; switch to pipeline or tensor model parallelism when a single GPU cannot hold parameters. Communications scale: AllReduce cost ~ O(log N) for ring/allreduce algorithms but depends on network bandwidth.
Interconnects: NVLink/NVSwitch and PCIe affect multi-GPU node performance; cross-node AllReduce is constrained by NIC bandwidth (e.g., 100 Gbps RoCE). Match topology-awareness in placement to reduce cross-node traffic.
Scheduler primitives: key scheduler features include gang scheduling, device plugins (k8s GPU device plugin / NVIDIA GPU Operator), preemption, priority classes, and node selectors/taints for GPU types. Gang scheduling ensures all job tasks start together to avoid stragglers.
Multi-tenancy & isolation: use MIG (A100+) for hardware partitioning, NVIDIA MPS for context multiplexing, or container-level limits for softer isolation. MIG gives stronger isolation and predictable VRAM/compute shares.
Preemptible/spot instances & checkpointing: when using preemptible GPUs, design frequent checkpointing and incremental state saves; employ elastic/resumable training libraries (framework checkpoint + requeue logic). Checkpoint frequency trades off runtime overhead vs lost work.
Fragmentation and packing: bin-packing increases utilization but may increase latency for large jobs; fragmentation metric = 1 − (sum idle_gpus / total_gpus). Defragment by backfill or allow preemption.
Autoscaling & cost model: tie autoscaler to queued job depth and utilization thresholds (e.g., scale up if queued > k jobs or gpu_util < 0.7); compute cost per effective GPU-hour = raw_cost / (utilization). Optimize batch sizes and mixed precision to lower cost-per-step.
Monitoring & SLOs: collect gpu_util, memory_used, memory_total, sm_efficiency, and job-level metrics (steps/sec, p95 latency). Define SLOs for throughput and job completion time; derive alerts from prometheus DCGM exporter.

Worked example — "Design a GPU scheduling strategy for a multi-tenant training cluster"

Frame: ask clarifying questions first — workload mix (short interactive jobs vs long batch training), SLA (fair-share or priority), toleration for preemption, allowed instance types (A100, V100), and whether checkpointing exists. Skeleton: (1) characterize workloads into classes (interactive, production retrain, hyperparameter sweep), (2) map classes to queues with priority and preemption rules, (3) choose scheduler primitives (gang scheduling for multi-node training, device plugin and node selectors for GPU types, MPS/MIG for small inference jobs), (4) pick autoscaling and checkpoint policy. Flag tradeoff: aggressive bin-packing and spot-instance usage improves cost but raises preemption and failure recovery complexity; prefer reserved capacity for high-priority training. Close by proposing measurable rollout: simulate scheduler using production traces, instrument DCGM and job completion metrics, and run a staged rollout with canary users. If more time: add fairness algorithms (DRF), scheduler simulator to tune backfill windows, and validate with adversarial workloads.

A second angle — "Optimize single-node multi‑GPU training throughput"

Here the focus shifts from scheduling to maximizing utilization on one machine. Start by profiling: overlap compute and communication by enabling NCCL asynchronous collectives and using gradient accumulation to increase compute per AllReduce. Use mixed precision to exploit tensor cores and reduce PCIe/NVLink pressure. If inter-GPU bandwidth is the bottleneck, prefer torch.distributed with nccl backend and enable /dev/shm staging for data loaders. Consider increasing per-GPU batch size until SM occupancy plateaus; if memory prevents this, use activation checkpointing. This framing emphasizes low-latency intra-node tricks rather than cluster-level allocation.

Common pitfalls

Pitfall: Optimizing for GPU count only. Many engineers request more GPUs without validating memory, interconnect, or IO, leading to poor utilization; always profile end-to-end (data loader → GPU compute → communication).

Pitfall: Ignoring preemption overhead. Assuming checkpointing is instantaneous underestimates lost work; quantify checkpoint time T_ckpt and requeue risk R to estimate expected wasted time = R * T_ckpt.

Pitfall: Overengineering scheduler features in interview. Propose a pragmatic incremental plan: classify workloads, implement priority queues, enable MIG/MPS, add checkpointing and monitoring, then iterate based on observed traces.

Connections

Interviewers often pivot to adjacent topics: distributed training algorithms (AllReduce vs parameter server), cost optimization (spot/commitment strategies), and data pipeline bottlenecks (sharded dataset serving, prefetching). Be ready to discuss how scheduling choices interact with dataset IO and model architecture.