Dynamic Batching for Synchronous Model Inference on One GPU with a 100-Input Cap
Company: Anthropic
Role: Product Manager
Category: System Design
Difficulty: medium
Interview Round: Other
Design the batching layer for a model server that runs inference on a single GPU. The GPU can process at most 100 inputs per batch, each caller submits one input and waits on its open request for that input's result, and the goal is to get as much work out of the GPU as its compute allows.
```hint Decide when a batch starts
Picture the first request that arrives during a quiet stretch. Ask what it experiences if the server only runs a batch once 100 inputs are waiting, and what else could decide when a batch goes to the GPU.
```
### Clarifying Questions
- What end-to-end latency can a waiting caller tolerate, and do clients enforce a hard timeout?
- Are the inputs fixed-size (for example, feature vectors or resized images), or variable-length sequences such as prompts that need padding or token-by-token generation?
- Is traffic steady, or does it come in bursts that exceed what one GPU can process?
- Are all callers treated equally, or do some requests carry a priority or a tighter deadline?
### What a Strong Answer Covers
- A batch launch policy that weighs the 100-input cap against how long callers wait, and how it behaves at low, moderate and high load.
- How each synchronous caller is held, matched to its own output, and released on success, error, timeout or disconnect.
- Ways to raise utilization beyond batch size, such as preparing the next batch while the current one runs and limiting padding waste.
- Overload behavior: a bounded queue, admission control and fast rejections when arrivals exceed what the GPU can serve.
- Isolation of failing inputs or batches, and the metrics that show whether utilization and latency goals are being met.
### Follow-up Questions
- The inputs turn out to be prompts for text generation, and outputs finish at very different lengths. How does your batching approach change?
- One malformed input makes the whole batch fail. How do you keep the other callers in that batch from getting errors?
- Traffic outgrows one GPU. What changes in the design, and what stays local to each GPU?
Overview: A system design prompt about serving model inference on one GPU that runs batches of up to 100 inputs while each caller waits for its own result. It tests batch launch policy, the trade-off between utilization and latency, overload control and isolating failures for synchronous requests.