Schedule Large and Small Model Inference on an Eight-GPU Pool

Schedule Large and Small Model Inference on an Eight-GPU Pool

Company: Anthropic

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Technical Screen

Overview: Schedule one-GPU small-model batches and eight-GPU large-model batches on a shared pool using cache-safe batching, deadline-aware queues, gang reservations, fairness, backpressure, overload handling, and workload simulation.