Design an Autoscaling LLM Inference Serving System
Company: Crusoe
Role: Software Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
Design an inference serving system for large language models that scales automatically: it adds model replicas when demand rises and removes them when demand falls, while meeting latency targets and keeping GPU cost under control. The report gives only this one-line prompt, so the models, scale and latency targets are open: settle them with the interviewer before designing.
```hint Pick the signal
GPU utilization and CPU load are poor triggers here. Think about which measurements show that requests are about to miss their latency targets before they actually do.
```
```hint Scaling up is slow
Estimate how long a new replica takes from the decision to scale until it serves its first request, and design around that delay.
```
### Clarifying Questions
- How many models, of what sizes, and does one replica need one GPU or several?
- What does traffic look like: steady, daily cycles, sudden spikes? Interactive chat, batch jobs, or both?
- What are the latency targets for time to first token and time per output token, and at which percentile?
- Is GPU capacity effectively unlimited from a cloud provider, or a fixed pool shared with other workloads?
- Are prefill and decode served on the same replicas or disaggregated into separate pools?
### What a Strong Answer Covers
- Requirements, latency targets and rough GPU-count estimates agreed first
- Routing and load balancing that account for queue length, prompt size and KV cache, not only round robin
- Scaling signals tied to the latency targets (queue time, KV cache usage, token throughput), with hysteresis and cooldowns
- Handling the slow cold start: weight loading, warm capacity, predictive scaling, and admission control while capacity arrives
- Safe scale-down that drains long-running streaming requests
- Multi-model serving, GPU capacity limits, failure handling, cost and observability
### Follow-up Questions
- Traffic triples within a minute, but new replicas need several minutes to start. What happens to requests in the meantime?
- How would you scale prefill and decode servers independently in a disaggregated deployment?
- How do you serve a long tail of rarely used models without keeping a GPU warm for each?
- How do you prevent oscillation, where the system keeps adding and removing replicas?
Overview: Design an LLM inference serving system that adds and removes model replicas automatically as traffic changes while meeting latency targets and controlling GPU cost. It tests choosing scaling signals, handling slow cold starts, load-aware routing, safe scale-down, multi-model serving and graceful behavior under overload.
Read the full Crusoe Software Engineer interview experience this question came from