Schedule Large and Small Model Inference on an Eight-GPU Pool
Company: Anthropic
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Technical Screen
Quick Answer: Schedule one-GPU small-model batches and eight-GPU large-model batches on a shared pool using cache-safe batching, deadline-aware queues, gang reservations, fairness, backpressure, overload handling, and workload simulation.