Explain Decoder-Only LLM Architecture and How vLLM Serves It Efficiently

This question comes from an AI-infrastructure-focused software engineering loop in which both the machine learning round and the system design round centered on the architecture of large language models (LLMs) and on common knowledge about vLLM, the open-source LLM inference and serving engine. Answer conversationally, as you would at a whiteboard, but be precise. ### Clarifying Questions - Should the answer target a generic modern decoder-only model, or a specific model family and size? - Is the serving setup a single GPU or a multi-GPU node? - Does the interviewer want inference-time behavior only, or also how the architecture is trained? ### Part 1 — Decoder-only LLM architecture Walk through what happens inside a modern decoder-only transformer LLM, from the input token IDs to the probability distribution over the next token. Describe the components of one transformer block, explain how the model knows the order of the tokens and why it cannot look at future tokens, and say roughly where the parameters and the compute are concentrated. Then explain what the key-value (KV) cache is, why generation needs it, and how to estimate its memory footprint for a given model and sequence length. ```hint Follow one position through a block Trace the hidden vector of a single token through one block and note which operations mix information across positions and which act on each position independently; the KV cache follows from that distinction. ``` #### What This Part Should Cover - The end-to-end path: tokens, embeddings, stacked blocks, final normalization, output projection and sampling - Block internals: causal self-attention, the feed-forward network, residual connections and normalization, plus how positional information enters - Where parameters and floating-point operations live, and how attention cost grows with sequence length - The KV cache: what it stores, why it avoids recomputation, and a formula for its size ### Part 2 — How vLLM serves these models efficiently Explain the main ideas that let vLLM achieve high throughput when serving such a model to many concurrent users: how it manages KV cache memory, how it batches requests of different lengths that arrive and finish at different times, and what it does when GPU memory runs out in the middle of generation. Finish by describing which settings you would examine first if a vLLM deployment showed low throughput or ran out of memory, and how you would serve a model that does not fit on one GPU. ```hint Start from the waste Consider how much KV cache memory a server wastes if it reserves one contiguous region per request, sized for the longest output that request might produce. ``` #### What This Part Should Cover - The prefill and decode phases, and why decode throughput is limited by memory bandwidth rather than arithmetic - Paged KV cache management: fixed-size blocks, per-sequence block tables and sharing - Continuous, iteration-level batching and how the scheduler handles memory pressure - Practical configuration and parallelism choices, with their trade-offs ### What a Strong Answer Covers - Architecture details tied to their serving cost, such as KV cache size limiting how many sequences can run at once - Quantitative reasoning with a sizing formula rather than vague statements - Accurate descriptions of vLLM mechanisms without overclaiming what they do - Explicit latency-versus-throughput and memory-versus-recomputation trade-offs - Awareness of failure modes: out-of-memory errors, repeated preemption, and long prompts delaying everyone else ### Follow-up Questions - How do multi-query and grouped-query attention change the KV cache size, and what do they cost? - How does speculative decoding speed up generation, and when does it fail to help? - When would you choose tensor parallelism, pipeline parallelism, or simply more model replicas? - How does automatic prefix caching work in vLLM, and which workloads benefit most from it?