LLM Inference Optimization Interview Questions: Quantization, Batching, KV Cache, and Latency

LLM inference optimization interviews are rarely a memory test about framework flags. They ask whether you can identify the bottleneck, choose a lever, predict the trade-off, and prove the result with a controlled benchmark. A good answer connects quantization, batching, KV-cache capacity, and latency metrics instead of claiming that one technique makes every model faster.

Start with the user-facing objective. A chat product may care about time to first token (TTFT) and smooth token streaming. An offline summarization job may care about total tokens per second and cost per request. A multi-tenant API also needs admission control, fairness, quality checks, and a safe rollback path.

Use PracHub’s machine-learning interview questions to rehearse the reasoning, but treat the model, GPU, engine version, traffic mix, and service-level objective in the interview prompt as the real constraints.

Quick answer: which optimization lever should you choose?

Do not “always quantize” or “always increase the batch.” Classify the workload: prompt-heavy or generation-heavy, short or long context, steady or bursty traffic, latency-sensitive or throughput-oriented.

Question 1: How do you define and measure inference latency?

Separate the request path into queueing, prefill, and decode. Prefill processes the prompt and builds the initial KV state. Decode generates output tokens one step at a time while reading that state. A useful mental model is:

TTFT ≈ queue time + prefill time + first decode step

end-to-end latency ≈ TTFT + decode time

TPOT (time per output token) and ITL (inter-token latency) describe the gaps during streaming. A service can have excellent total throughput and still fail a chat SLO if queued requests have high p99 TTFT. Conversely, an offline batch can tolerate higher TTFT if it maximizes tokens per second.

vLLM exposes queue, prefill, decode, TTFT, inter-token, end-to-end, prompt-token, generation-token, request-state, and KV-cache metrics. Its benchmark documentation also warns that metric names and measurement points vary, so state exactly where the timer starts and ends. Report p50 and p95/p99, not only an average.

What a strong answer sounds like

“I would segment by prompt length, output length, replica, and concurrency. Rising TTFT with stable TPOT points to queue or prefill pressure; rising TPOT points to decode, memory bandwidth, kernels, or communication.”

Question 2: What does quantization buy you, and what can it break?

Quantization represents weights, activations, or cached values with fewer bits. Weight-only formats such as W4A16 store weights at four bits while keeping activations in a higher precision. W8A8 quantizes both weights and activations and needs compatible kernels and calibration. The correct choice depends on hardware support, model architecture, latency target, and acceptable quality change.

Hugging Face documents 8-bit loading with bitsandbytes as roughly halving weight memory for supported models; its 4-bit path can reduce memory further, but metadata, unquantized modules, and runtime buffers still consume memory. vLLM lists formats such as AWQ, GPTQ, BitsAndBytes, FP8, and quantized KV cache, while noting that support varies by hardware. That is why “INT4 is faster” is an incomplete answer.

Quantization can help when weight memory or memory bandwidth is the bottleneck. It can fail to help when the workload is dominated by tokenization, queueing, communication, or a kernel that dequantizes values inefficiently. A lower-precision checkpoint can also change logit rankings, long-context behavior, tool-call syntax, or safety refusals. Evaluate a representative quality suite, not only perplexity on one corpus.

A practical quantization experiment

Keep the model, tokenizer, prompts, sampling parameters, batch limits, and GPU fixed. Compare the baseline and quantized variants on:

1. output quality by task slice and failure category;
2. peak memory and admission capacity;
3. TTFT and TPOT/ITL at the same concurrency;
4. tokens per second and cost per completed request; and
5. error rate, timeouts, and p95/p99 behavior.

If the quantized model fits twice as many requests but quality drops on long-context retrieval, the result is a product trade-off, not an unconditional win.

Question 3: How is KV-cache memory different from weight memory?

Weights are loaded once per worker. The KV cache grows with active sequences and their cached tokens. A useful approximation for a decoder-only model is:

KV bytes ≈ tokens × layers × 2 × KV heads × head dimension × bytes per element

The factor of two is for keys and values. Grouped-query or multi-query attention can reduce KV heads relative to query heads. Multiply by active requests and account for block metadata, alignment, temporary buffers, tensor parallelism, and partially filled blocks.

This distinction changes the diagnosis. If the model weights do not fit, quantize or shard the weights. If weights fit but long contexts cause preemptions or rejected requests, KV-cache capacity and scheduling are the likely constraints. Increasing batch size without estimating KV demand can convert a throughput experiment into an out-of-memory incident.

Question 4: Why do engines use paged KV caches and continuous batching?

An autoregressive request does not reveal its final output length. Contiguous allocation either reserves too much space or repeatedly moves state as the sequence grows. Paged KV designs split logical token ranges into fixed-size blocks mapped to physical blocks in a pool, reducing fragmentation and allowing on-demand allocation.

That memory layout is separate from continuous or in-flight batching. A scheduler can add, finish, or cancel sequences between iterations. TensorRT-LLM describes in-flight batching as interleaving context (prefill) and generation (decode); the gain is utilization, while long prefills can still trade TTFT against TPOT.

In an interview, state the policy: maximum scheduled tokens, maximum active sequences, fairness or priority, cancellation, and what happens when the cache cannot admit another request. “Use continuous batching” is not a capacity plan without those limits.

Question 5: When does prefix caching help?

Prefix caching reuses KV blocks when a new request shares a completed prefix with an earlier request. Good examples include a long system prompt, a repeated document, or later turns in the same conversation. vLLM’s documentation is explicit about the boundary: prefix caching can skip repeated prefill computation, but it does not make new output-token decoding cheaper. A workload with unique prompts or very long answers may see little benefit.

Ask about correctness and isolation. The cache key must separate model and tokenizer versions, adapters, multimodal inputs, and any other state that changes the KV values. In multi-tenant systems, a shared prefix can create a timing side channel because cache hits change TTFT. vLLM documents request-level cache salting as an isolation control; it reduces reuse across trust boundaries in exchange for privacy.

Question 6: Should you quantize the KV cache?

KV-cache quantization can increase the number of cached tokens that fit in GPU memory. Hugging Face documents quantized-cache backends and warns that quantizing the cache can hurt latency when contexts are short and there is enough memory without it. TensorRT-LLM supports INT8 or FP8 KV caches and shows workload-specific throughput results, while also warning that aggressive quantization can degrade output quality.

The interview answer should be conditional: use it when KV memory limits concurrency or long context, verify that the attention kernels support the dtype, and benchmark quality plus TPOT. Do not use a weight-quantization result as evidence that KV-cache quantization will behave the same way; the values are produced and consumed at different points in the computation.

Question 7: How would you diagnose a latency regression?

Start with a before-and-after trace under the same prompt and output distributions. Use a table like this:

Change one variable at a time. For a queueing regression, lowering the token budget or adding admission control may improve TTFT while reducing peak throughput. For a kernel regression, a different attention backend or GPU may help, but that is a version- and hardware-specific experiment. Define a rollback threshold before the canary starts.

Question 8: How do you design a credible benchmark?

Build a workload matrix instead of one “average” prompt: short and long prompts and outputs, shared and unique prefixes, low and high concurrency, and representative tenant mix. Decide whether the prefix cache is warm and record model revision, tokenizer, quantization recipe, GPU, engine, batch limits, and sampling settings.

Measure quality and systems behavior together. Tokens per second alone can hide worse p99 TTFT or output changes. Compare at the same quality target and repeat close results across trials.

Eight-step answer framework

1. State the SLO: TTFT, TPOT/ITL, throughput, cost, or a combination.
2. Classify the workload by prompt length, output length, concurrency, and prefix reuse.
3. Estimate weight and KV-cache memory separately.
4. Identify the bottleneck from queue, prefill, decode, memory, and communication metrics.
5. Select one lever: quantization, batching, cache policy, kernel, or parallelism.
6. Name the quality, fairness, isolation, and operational risks.
7. Propose a fixed workload benchmark and a canary comparison.
8. Define success, rollback, and the next hypothesis before changing variables.

Practice with five verified PracHub questions

These records help you rehearse the underlying systems reasoning. They are practice material, not predictions of an employer’s exact LLM interview questions.

Common interview mistakes

• Treating TTFT, TPOT, and total latency as interchangeable.
• Quoting a throughput improvement without specifying model, hardware, workload, and quality target.
• Saying quantization only changes memory and never affects output quality or kernel choice.
• Treating KV-cache paging as the same thing as continuous batching.
• Assuming prefix caching speeds decode when it primarily skips repeated prefill.
• Increasing concurrency until the cache is full without an admission or cancellation policy.

Frequently asked questions

Is lower precision always faster?

No. It can reduce memory traffic or make a model fit, but unsupported kernels, dequantization, synchronization, or quality safeguards can remove the gain. Benchmark the exact engine and hardware.

What is usually the first bottleneck in long-context serving?

It may be KV-cache capacity, prefill compute, or queueing. Measure prompt length, cache usage, prefill time, waiting requests, and preemptions before choosing a fix.

Does prefix caching reduce all inference cost?

No. It avoids repeated work for a shared prompt prefix. New output-token decoding still runs, and unique prompts have no reusable prefix.

Should I optimize for throughput or latency?

Choose from the product SLO. Interactive chat usually needs TTFT and streaming smoothness; offline jobs often prioritize total throughput and cost. Report both when traffic mixes the two.

Sources and Further Reading

• vLLM: Quantization
• vLLM: Paged Attention
• vLLM: Automatic Prefix Caching
• vLLM: Production Metrics
• vLLM: Benchmark CLI and latency definitions
• Hugging Face: bitsandbytes quantization
• Hugging Face: KV cache strategies
• NVIDIA TensorRT-LLM: in-flight batching and attention
• NVIDIA TensorRT-LLM: KV cache system
• NVIDIA TensorRT-LLM: FP8 quantization
• Efficient Memory Management for Large Language Model Serving with PagedAttention

Research note: checked September 6, 2026. Engine defaults, quantization support, metrics, kernels, and performance change by release and hardware. Verify the documentation and benchmark the exact deployment before making a production decision.