Design a batched inference API
Company: Anthropic
Role: Software Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Onsite
Design an online machine learning inference service that supports dynamic batching.
Multiple clients send small synchronous prediction requests to an API. Running each request individually wastes GPU capacity, so the system should combine compatible requests into batches before model execution. At the same time, the service must still meet a latency SLO for online traffic.
Discuss:
- The external API and request/response schema
- How requests are grouped into compatible batches
- Queueing and scheduling logic, including max batch size and max wait time
- Handling variable input sizes or sequence lengths
- Model routing and versioning
- Timeouts, cancellations, and partial failures
- Autoscaling, reliability, and observability
- Trade-offs between latency, throughput, and cost
Quick Answer: This question evaluates competency in designing scalable, low-latency ML inference systems with dynamic batching, covering system architecture, request batching and scheduling, model routing/versioning, and operational concerns such as autoscaling, reliability, timeouts, and observability.