ML Breadth: Bias-Variance, Normalization, Transformer Attention and the KV Cache

This machine learning breadth round is a sequence of short conceptual questions: the bias–variance trade-off, normalization, the transformer architecture, attention, and the KV cache. Answer each part the way you would aloud in the round: a precise explanation, the reasoning or math behind it, and how it affects a practical decision. ### Clarifying Questions - For normalization, does the interviewer mean scaling the input features, normalization layers inside a network (batch, layer or RMS normalization), or both? - Should the transformer discussion center on decoder-only language models or on the original encoder–decoder model? - How much math is expected: a derivation, or the key formulas with intuition? ### Part 1 — Bias–variance trade-off Explain the bias–variance trade-off. How does it show up in training and validation error, and what would you do about a model with high bias versus one with high variance? ```hint Decompose the error Write the expected squared error of a prediction at one input, averaged over the training sets you could have drawn, as a sum of terms, and ask which terms depend on the particular training set. ``` #### What This Part Should Cover - Definitions of bias, variance and irreducible noise, and how they add up for squared error - Diagnosis from training and validation error or from learning curves - Remedies for each case, and how model capacity and regularization move a model along the trade-off ### Part 2 — Normalization Why do we normalize? Compare batch normalization and layer normalization: what each computes, how each behaves at training and inference time, and which one transformers use and why. ```hint Over which axes For activations shaped batch × sequence × features, ask over which axes each method computes its mean and variance, and what that implies for a batch of one or for sequences of different lengths. ``` #### What This Part Should Cover - Why scaling inputs and normalizing activations helps optimization - How batch and layer normalization are computed, including the learned scale and shift and the behavior at inference - Why transformers use layer normalization or its variants, and where the normalization sits in a block ### Part 3 — Transformer architecture and attention Walk through a transformer block. Explain scaled dot-product attention: where the queries, keys and values come from, why the scores are scaled, and how masking works. Why use several attention heads, and what does attention cost as the sequence grows? ```hint Follow one token Trace one token's vector through a block, noting the shape at each step, what the attention weights are normalized over, and where the residual paths join. ``` #### What This Part Should Cover - The components of a block: attention, feed-forward network, residual connections, normalization, and positional information - The attention formula, the reason for the scaling factor, and causal masking - Multi-head attention, and the time and memory cost as a function of sequence length ### Part 4 — KV cache What is the KV cache, why does it speed up autoregressive generation, and what does it cost? How do you reduce that cost? ```hint What repeats at each step When generating the next token, list which key and value vectors were already computed for earlier tokens, and whether the new token can change them. ``` #### What This Part Should Cover - What is cached, at which layers, and why caching is valid in a causal decoder - The compute saved, and how the prompt-processing and token-generation phases differ - How cache memory scales with layers, heads, sequence length and batch size, and techniques that reduce it ### What a Strong Answer Covers - Precise definitions, with formulas where they clarify, and exact tensor shapes rather than hand-waving - Connects each concept to a practical decision: diagnosing a model, choosing a normalization, sizing inference memory - Reasons about cost in terms of sequence length, model width and batch size - Knows the limits of the textbook picture, for example how very large models complicate the classic bias–variance curve ### Follow-up Questions - How do multi-query and grouped-query attention change the KV cache size, and what do they cost in model quality? - Why does batch normalization behave differently at training and inference time, and what goes wrong with very small training batches? - How does the KV cache affect serving many requests of different lengths at once? - Large over-parameterized networks can fit their training data perfectly and still generalize well. How does that fit the bias–variance picture?