Debug an LLM Inference Bottleneck and a Prefill Latency Spike at 23 to 24 QPS

Quick Overview

Debug an LLM inference service that misses its latency or throughput targets, then investigate a prefill server whose latency spikes when traffic rises from 23 to 24 QPS in a prefill/decode disaggregated deployment. It tests latency decomposition, queueing intuition near saturation and disaggregation-specific failure points.

Debug an LLM Inference Bottleneck and a Prefill Latency Spike at 23 to 24 QPS

Company: Crusoe

Role: Software Engineer

Category: ML System Design

Difficulty: medium

Interview Round: Onsite

Two debugging questions about a large language model (LLM) inference service. Answer them as an investigation: what you would measure, in what order, and what each result would tell you. ### Clarifying Questions - Which latency targets apply (time to first token, time per output token, at which percentile), and what do the prompt and output lengths look like? - What observability exists today: engine metrics, per-request traces, GPU profilers? ### Part 1 — Find the bottleneck An inference system has a bottleneck: it is missing its latency or throughput targets. How do you debug it and find where the bottleneck is? ```hint Split the latency Decide which stages a request's end-to-end latency breaks into, and which measurement would show that one particular stage is the one growing. ``` #### What This Part Should Cover - Decomposing latency into queueing, prefill, decode and the layers around the engine - The metrics and tools used at each layer: engine metrics, GPU compute and memory signals, profilers - Telling compute, memory-bandwidth, KV-capacity, CPU and network limits apart - A controlled load test that reproduces the problem ### Part 2 — A prefill latency spike between 23 and 24 QPS The service separates prefill and decode onto different servers (prefill/decode disaggregation): prefill servers process prompts and hand each request's KV cache to decode servers, which generate the output tokens. A prefill server's latency is normal at 23 QPS, but when traffic rises to 24 QPS its latency spikes. How do you debug this? ```hint A small step, a big jump Ask what kind of system answers a load increase of about 4 percent with a large jump in latency, and what you would plot to confirm it. ``` #### Clarifying Questions for this Part - Is "latency" the time to first token seen by the client, the prefill compute time, or the time until the KV cache reaches the decode server? - Did the request mix (prompt lengths, prefix-cache hit rate) stay the same as traffic rose? - At 24 QPS, does latency settle at a higher level or keep climbing for as long as the load lasts? #### What This Part Should Cover - Recognizing a capacity knee and checking it with queue depth and throughput against offered load - Separating queue wait from prefill compute and from the KV hand-off to the decode side - Discrete thresholds that can cause a step change: per-step token budget, KV memory, decode-side backpressure, CPU-side limits - Confirming the cause with a load sweep, then fixing it and guarding against a repeat ### What a Strong Answer Covers - A hypothesis-driven method that measures before tuning anything - Signals chosen per layer, with a clear idea of what each pattern means - Queueing intuition: latency explodes as utilization approaches its limit, and tail latency moves first - Failure points specific to disaggregation: KV transfer, decode backpressure and the ratio of prefill to decode servers - A fix plus a guard, such as capacity headroom, admission control or alerts on queue depth ### Follow-up Questions - How would you find the maximum QPS a prefill server can sustain within its latency target before production traffic reaches it? - The spike appears only at certain hours although QPS is the same. What would you look at? - How do you decide the ratio of prefill servers to decode servers, and when would you change it?

Overview: Debug an LLM inference service that misses its latency or throughput targets, then investigate a prefill server whose latency spikes when traffic rises from 23 to 24 QPS in a prefill/decode disaggregated deployment. It tests latency decomposition, queueing intuition near saturation and disaggregation-specific failure points.

Read the full Crusoe Software Engineer interview experience this question came from

Crusoe logo
Crusoe
Nov 2, 2026
mediumSoftware EngineerOnsiteML System Design
1
0

Two debugging questions about a large language model (LLM) inference service. Answer them as an investigation: what you would measure, in what order, and what each result would tell you.

Clarifying Questions Guidance

  • Which latency targets apply (time to first token, time per output token, at which percentile), and what do the prompt and output lengths look like?
  • What observability exists today: engine metrics, per-request traces, GPU profilers?

Part 1 — Find the bottleneck

An inference system has a bottleneck: it is missing its latency or throughput targets. How do you debug it and find where the bottleneck is?

What This Part Should Cover Guidance

  • Decomposing latency into queueing, prefill, decode and the layers around the engine
  • The metrics and tools used at each layer: engine metrics, GPU compute and memory signals, profilers
  • Telling compute, memory-bandwidth, KV-capacity, CPU and network limits apart
  • A controlled load test that reproduces the problem

Part 2 — A prefill latency spike between 23 and 24 QPS

The service separates prefill and decode onto different servers (prefill/decode disaggregation): prefill servers process prompts and hand each request's KV cache to decode servers, which generate the output tokens. A prefill server's latency is normal at 23 QPS, but when traffic rises to 24 QPS its latency spikes. How do you debug this?

Clarifying Questions for this Part Guidance

  • Is "latency" the time to first token seen by the client, the prefill compute time, or the time until the KV cache reaches the decode server?
  • Did the request mix (prompt lengths, prefix-cache hit rate) stay the same as traffic rose?
  • At 24 QPS, does latency settle at a higher level or keep climbing for as long as the load lasts?

What This Part Should Cover Guidance

  • Recognizing a capacity knee and checking it with queue depth and throughput against offered load
  • Separating queue wait from prefill compute and from the KV hand-off to the decode side
  • Discrete thresholds that can cause a step change: per-step token budget, KV memory, decode-side backpressure, CPU-side limits
  • Confirming the cause with a load sweep, then fixing it and guarding against a repeat

What a Strong Answer Covers Guidance

  • A hypothesis-driven method that measures before tuning anything
  • Signals chosen per layer, with a clear idea of what each pattern means
  • Queueing intuition: latency explodes as utilization approaches its limit, and tail latency moves first
  • Failure points specific to disaggregation: KV transfer, decode backpressure and the ratio of prefill to decode servers
  • A fix plus a guard, such as capacity headroom, admission control or alerts on queue depth

Follow-up Questions Guidance

  • How would you find the maximum QPS a prefill server can sustain within its latency target before production traffic reaches it?
  • The spike appears only at certain hours although QPS is the same. What would you look at?
  • How do you decide the ratio of prefill servers to decode servers, and when would you change it?