ML Basics Check: Bayes' Rule, Forward-Pass Arithmetic, KNN and Transformer Concepts

An online assessment for a machine learning engineering internship lasts 90 minutes and has six questions: four multiple-choice questions and two coding questions. The multiple-choice questions were basic, and the candidate recalled only their topics: - applying Bayes' rule; - simple calculations of a neuron's output or a network's parameters in a forward pass; - basic concept checks about k-nearest neighbors (KNN) and about Transformers. The practice items below cover those topics. Their numbers and statements are illustrative, not the original items. Answer each part, showing enough working that you could pick the right option quickly under time pressure. ### Clarifying Questions - Does each multiple-choice item have exactly one correct option, or can several options be correct? - Is a calculator allowed, or are the calculations expected by hand? - Are wrong answers penalized, so that leaving an item blank can be better than guessing? ### Part 1 — Bayes' rule A detector flags suspicious items. 2% of items are truly suspicious. The detector flags 90% of suspicious items and 5% of normal items. 1. An item is flagged. What is the probability that it is truly suspicious? 2. A second detector with the same rates, whose errors are independent of the first detector's once the item's true class is known, also flags the item. What is the probability now? ```hint Count a population Picture 10,000 items and count how many land in each of the four groups: suspicious or normal, flagged or not flagged. ``` #### What This Part Should Cover - Correct identification of the prior, the two likelihoods and the overall flag rate - A correct posterior, with the effect of the low base rate explained - How a second, conditionally independent observation updates the posterior ### Part 2 — Forward-pass arithmetic and parameter counts 1. A single neuron has weights `(0.5, -1.0, 2.0)` and bias `0.1`, and receives the input `(2, 1, 0.5)`. What is its output with a ReLU activation? With a sigmoid activation? 2. A fully connected network maps 4 input features to a hidden layer of 3 units, then to 2 output units. Every layer has biases. How many trainable parameters does it have? 3. A network has two inputs, two ReLU hidden units and one linear output. Hidden unit 1 has weights `(1, -1)` and bias `0`. Hidden unit 2 has weights `(0.5, 0.5)` and bias `-1`. The output is `2 * h1 - 1 * h2 + 0.5`. For the input `(1, 2)`, compute both hidden activations and the output. Which of the network's parameters receive a zero gradient from this example during backpropagation? ```hint Pre-activation first Compute each unit's weighted sum plus bias before applying its activation. For parameter counts, remember that every unit in a layer has its own bias. ``` #### What This Part Should Cover - Correct pre-activation and activation values - Parameter counts derived from the layer shapes - What the inactive region of ReLU means for the gradients of the parameters around it ### Part 3 — KNN concept checks 1. Mark each statement true or false, with a one-sentence reason. - (a) KNN has almost no training cost, but its prediction cost grows with the size of the training set. - (b) Increasing k always lowers the test error. - (c) With k = 1, training accuracy is 100%, as long as no two identical training points have different labels. - (d) Rescaling one feature, for example from meters to millimeters, does not change KNN's predictions. - (e) KNN works as well with 1,000 features as with 10, given the same number of training points. - (f) KNN can be used for regression by averaging the neighbors' target values. 2. One-dimensional training points: class A at 0, 1 and 2; class B at 2.6 and 5. Using absolute distance and a majority vote, predict the class of a query at 2.4 with k = 1 and with k = 3. ```hint What the model stores KNN keeps the training set and does all of its work at prediction time. Judge each statement by what that means for cost, for the distances between points, and for small versus large k. ``` #### What This Part Should Cover - Correct verdicts, each with the mechanism behind it - The bias-variance effect of k, and why distances depend on feature scale and dimension - The worked prediction for both values of k ### Part 4 — Transformer concept checks 1. Mark each statement true or false, with a one-sentence reason. - (a) In standard self-attention, compute and memory grow quadratically with the sequence length. - (b) Without positional encodings and without an attention mask, self-attention is permutation-equivariant: reordering the input tokens only reorders the outputs. - (c) Attention scores are divided by $\sqrt{d_k}$ mainly to reduce the number of parameters. - (d) In a decoder used for text generation, a causal mask stops each position from attending to later positions. - (e) Using h attention heads multiplies the parameter count of the attention projections by h, compared with a single head at the same model dimension. - (f) Layer normalization normalizes each feature across the examples in a batch. 2. A multi-head attention layer has model dimension $d_{\text{model}} = 512$ and 8 heads, with a bias on each of the query, key, value and output projections. What is each head's dimension, and how many parameters do the four projections have in total? ```hint Write down the shapes Write the shapes of the query, key and value matrices and of the attention-score matrix for a sequence of n tokens. Most of these statements follow from those shapes and from how softmax behaves on large inputs. ``` #### What This Part Should Cover - Correct verdicts grounded in how attention is computed - The roles of scaling, masking and positional information - Parameter arithmetic derived from the projection shapes ### What a Strong Answer Covers - Correct final values with the intermediate quantities shown, so arithmetic slips can be caught - A reason for each true or false verdict that names the mechanism, not a bare verdict - Quick sanity checks, such as probabilities between 0 and 1 and parameter counts that follow from shapes, which make eliminating options fast - A pace that leaves most of the 90 minutes for the two coding questions ### Follow-up Questions - In Part 1, how does the answer to question 1 change if the base rate rises from 2% to 20%, and why do detectors for rare events raise so many false alarms? - In Part 2, question 3, what is the gradient of the output with respect to hidden unit 2's bias, and what is the "dying ReLU" problem? - How would you make KNN prediction fast on tens of millions of points, and what would you give up? - How do efficient-attention methods reduce the quadratic cost in sequence length, and what do they trade away?