LLM Inference Basics: Prefill vs Decode and an Engine's Request Lifecycle
Company: Crusoe
Role: Software Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Onsite
This round tested the core concepts of serving large language models (LLMs). Answer both parts as you would to an interviewer who works on inference engines: explain the mechanisms, not only their names.
### Clarifying Questions
- Should the answer assume a decoder-only transformer on GPUs, and one GPU per model replica or a model sharded across several GPUs?
- Should Part 2 describe a particular kind of engine, for example one with continuous batching and a paged KV cache, or a generic one?
### Part 1 — Prefill versus decode
Explain how the prefill phase and the decode phase of LLM inference differ: what each phase computes, what limits its speed on a GPU, and which latency the user experiences from each.
```hint Count the work per pass
Compare how many tokens each phase pushes through the model in one forward pass, and what has to be read from GPU memory for that pass.
```
#### What This Part Should Cover
- What each phase computes, and how each one writes to and reads from the KV cache
- Compute-bound versus memory-bandwidth-bound behavior, and why batching affects the two phases differently
- The latency metrics each phase drives: time to first token and time per output token
- Consequences for scheduling or deployment, such as chunked prefill or running the phases on separate servers
### Part 2 — How an inference engine processes a request
Walk through what happens inside an LLM inference engine from the moment a generation request arrives until the last token reaches the client.
```hint Follow one prompt
Trace a single request through the engine and ask which component owns it at each step, and what GPU memory it holds while it waits for its next token.
```
#### What This Part Should Cover
- Intake: validation, tokenization, admission and the scheduler's waiting queue
- Continuous batching and how prefill and decode work share each engine step
- KV cache memory management: allocation, prefix reuse, and preemption when memory runs out
- Sampling, stop conditions, streaming the detokenized output, and releasing resources
### What a Strong Answer Covers
- Explains the limits through arithmetic intensity: how much computation each byte read from GPU memory supports
- Ties each phase to time to first token and time per output token, and to the throughput-versus-latency tension of batching
- Treats KV cache memory as the resource that bounds concurrency
- Describes the engine's behavior under load: queueing, long prompts delaying others, and preemption
### Follow-up Questions
- Why does a larger batch raise decode throughput almost for free up to a point, and what ends that?
- How does speculative decoding change the decode phase, and when does it stop helping?
- What does prefix caching save, and for which workloads does it pay off?
- Why might a team run prefill and decode on different servers, and what new cost does that introduce?
Overview: Explain how the prefill and decode phases of large language model inference differ and walk through how an inference engine handles a request from arrival to the last streamed token. It tests understanding of compute versus memory-bandwidth limits, KV cache management, continuous batching and latency metrics.
Read the full Crusoe Software Engineer interview experience this question came from