Design a Dynamically Batched Inference API
Company: Anthropic
Role: Machine Learning Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
Overview: Design a high-concurrency language-model inference API that dynamically batches compatible requests on a limited GPU pool. Balance interactive latency and throughput through admission control, memory-aware scheduling, continuous batching, cancellation, streaming, fairness, and overload handling.
Read the full Anthropic Machine Learning Engineer interview experience this question came from