LLM Inference Basics: Prefill vs Decode and an Engine's Request Lifecycle

Quick Overview

Explain how the prefill and decode phases of large language model inference differ and walk through how an inference engine handles a request from arrival to the last streamed token. It tests understanding of compute versus memory-bandwidth limits, KV cache management, continuous batching and latency metrics.

LLM Inference Basics: Prefill vs Decode and an Engine's Request Lifecycle

Company: Crusoe

Role: Software Engineer

Category: Machine Learning

Difficulty: medium

Interview Round: Onsite

This round tested the core concepts of serving large language models (LLMs). Answer both parts as you would to an interviewer who works on inference engines: explain the mechanisms, not only their names. ### Clarifying Questions - Should the answer assume a decoder-only transformer on GPUs, and one GPU per model replica or a model sharded across several GPUs? - Should Part 2 describe a particular kind of engine, for example one with continuous batching and a paged KV cache, or a generic one? ### Part 1 — Prefill versus decode Explain how the prefill phase and the decode phase of LLM inference differ: what each phase computes, what limits its speed on a GPU, and which latency the user experiences from each. ```hint Count the work per pass Compare how many tokens each phase pushes through the model in one forward pass, and what has to be read from GPU memory for that pass. ``` #### What This Part Should Cover - What each phase computes, and how each one writes to and reads from the KV cache - Compute-bound versus memory-bandwidth-bound behavior, and why batching affects the two phases differently - The latency metrics each phase drives: time to first token and time per output token - Consequences for scheduling or deployment, such as chunked prefill or running the phases on separate servers ### Part 2 — How an inference engine processes a request Walk through what happens inside an LLM inference engine from the moment a generation request arrives until the last token reaches the client. ```hint Follow one prompt Trace a single request through the engine and ask which component owns it at each step, and what GPU memory it holds while it waits for its next token. ``` #### What This Part Should Cover - Intake: validation, tokenization, admission and the scheduler's waiting queue - Continuous batching and how prefill and decode work share each engine step - KV cache memory management: allocation, prefix reuse, and preemption when memory runs out - Sampling, stop conditions, streaming the detokenized output, and releasing resources ### What a Strong Answer Covers - Explains the limits through arithmetic intensity: how much computation each byte read from GPU memory supports - Ties each phase to time to first token and time per output token, and to the throughput-versus-latency tension of batching - Treats KV cache memory as the resource that bounds concurrency - Describes the engine's behavior under load: queueing, long prompts delaying others, and preemption ### Follow-up Questions - Why does a larger batch raise decode throughput almost for free up to a point, and what ends that? - How does speculative decoding change the decode phase, and when does it stop helping? - What does prefix caching save, and for which workloads does it pay off? - Why might a team run prefill and decode on different servers, and what new cost does that introduce?

Overview: Explain how the prefill and decode phases of large language model inference differ and walk through how an inference engine handles a request from arrival to the last streamed token. It tests understanding of compute versus memory-bandwidth limits, KV cache management, continuous batching and latency metrics.

Read the full Crusoe Software Engineer interview experience this question came from

Crusoe logo
Crusoe
Nov 2, 2026
mediumSoftware EngineerOnsiteMachine Learning
0
0

This round tested the core concepts of serving large language models (LLMs). Answer both parts as you would to an interviewer who works on inference engines: explain the mechanisms, not only their names.

Clarifying Questions Guidance

  • Should the answer assume a decoder-only transformer on GPUs, and one GPU per model replica or a model sharded across several GPUs?
  • Should Part 2 describe a particular kind of engine, for example one with continuous batching and a paged KV cache, or a generic one?

Part 1 — Prefill versus decode

Explain how the prefill phase and the decode phase of LLM inference differ: what each phase computes, what limits its speed on a GPU, and which latency the user experiences from each.

What This Part Should Cover Guidance

  • What each phase computes, and how each one writes to and reads from the KV cache
  • Compute-bound versus memory-bandwidth-bound behavior, and why batching affects the two phases differently
  • The latency metrics each phase drives: time to first token and time per output token
  • Consequences for scheduling or deployment, such as chunked prefill or running the phases on separate servers

Part 2 — How an inference engine processes a request

Walk through what happens inside an LLM inference engine from the moment a generation request arrives until the last token reaches the client.

What This Part Should Cover Guidance

  • Intake: validation, tokenization, admission and the scheduler's waiting queue
  • Continuous batching and how prefill and decode work share each engine step
  • KV cache memory management: allocation, prefix reuse, and preemption when memory runs out
  • Sampling, stop conditions, streaming the detokenized output, and releasing resources

What a Strong Answer Covers Guidance

  • Explains the limits through arithmetic intensity: how much computation each byte read from GPU memory supports
  • Ties each phase to time to first token and time per output token, and to the throughput-versus-latency tension of batching
  • Treats KV cache memory as the resource that bounds concurrency
  • Describes the engine's behavior under load: queueing, long prompts delaying others, and preemption

Follow-up Questions Guidance

  • Why does a larger batch raise decode throughput almost for free up to a point, and what ends that?
  • How does speculative decoding change the decode phase, and when does it stop helping?
  • What does prefix caching save, and for which workloads does it pay off?
  • Why might a team run prefill and decode on different servers, and what new cost does that introduce?