Debug an LLM Inference Bottleneck and a Prefill Latency Spike at 23 to 24 QPS
Company: Crusoe
Role: Software Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
Two debugging questions about a large language model (LLM) inference service. Answer them as an investigation: what you would measure, in what order, and what each result would tell you.
### Clarifying Questions
- Which latency targets apply (time to first token, time per output token, at which percentile), and what do the prompt and output lengths look like?
- What observability exists today: engine metrics, per-request traces, GPU profilers?
### Part 1 — Find the bottleneck
An inference system has a bottleneck: it is missing its latency or throughput targets. How do you debug it and find where the bottleneck is?
```hint Split the latency
Decide which stages a request's end-to-end latency breaks into, and which measurement would show that one particular stage is the one growing.
```
#### What This Part Should Cover
- Decomposing latency into queueing, prefill, decode and the layers around the engine
- The metrics and tools used at each layer: engine metrics, GPU compute and memory signals, profilers
- Telling compute, memory-bandwidth, KV-capacity, CPU and network limits apart
- A controlled load test that reproduces the problem
### Part 2 — A prefill latency spike between 23 and 24 QPS
The service separates prefill and decode onto different servers (prefill/decode disaggregation): prefill servers process prompts and hand each request's KV cache to decode servers, which generate the output tokens. A prefill server's latency is normal at 23 QPS, but when traffic rises to 24 QPS its latency spikes. How do you debug this?
```hint A small step, a big jump
Ask what kind of system answers a load increase of about 4 percent with a large jump in latency, and what you would plot to confirm it.
```
#### Clarifying Questions for this Part
- Is "latency" the time to first token seen by the client, the prefill compute time, or the time until the KV cache reaches the decode server?
- Did the request mix (prompt lengths, prefix-cache hit rate) stay the same as traffic rose?
- At 24 QPS, does latency settle at a higher level or keep climbing for as long as the load lasts?
#### What This Part Should Cover
- Recognizing a capacity knee and checking it with queue depth and throughput against offered load
- Separating queue wait from prefill compute and from the KV hand-off to the decode side
- Discrete thresholds that can cause a step change: per-step token budget, KV memory, decode-side backpressure, CPU-side limits
- Confirming the cause with a load sweep, then fixing it and guarding against a repeat
### What a Strong Answer Covers
- A hypothesis-driven method that measures before tuning anything
- Signals chosen per layer, with a clear idea of what each pattern means
- Queueing intuition: latency explodes as utilization approaches its limit, and tail latency moves first
- Failure points specific to disaggregation: KV transfer, decode backpressure and the ratio of prefill to decode servers
- A fix plus a guard, such as capacity headroom, admission control or alerts on queue depth
### Follow-up Questions
- How would you find the maximum QPS a prefill server can sustain within its latency target before production traffic reaches it?
- The spike appears only at certain hours although QPS is the same. What would you look at?
- How do you decide the ratio of prefill servers to decode servers, and when would you change it?
Overview: Debug an LLM inference service that misses its latency or throughput targets, then investigate a prefill server whose latency spikes when traffic rises from 23 to 24 QPS in a prefill/decode disaggregated deployment. It tests latency decomposition, queueing intuition near saturation and disaggregation-specific failure points.
Read the full Crusoe Software Engineer interview experience this question came from