## Machine Learning Fundamentals: Optimizers, Scaling Laws, and Clustering This is a fundamentals round covering three loosely related areas of machine learning. Answer each part precisely, with the math where it matters, and be ready to justify trade-offs and corner cases rather than just naming methods. ### Constraints & Assumptions - "Optimizers" refers to first-order gradient methods used to train deep neural networks. - "Scaling laws" refers to empirical power-law relationships between a model's loss and the resources used to train it (parameters, data, compute) for large language models. - The clustering part assumes unlabeled data and asks you to compare two classical unsupervised methods. - You may use standard notation; define your symbols. ### Clarifying Questions to Ask - For optimizers: are we discussing training stability of large transformers specifically, or general deep-network training? - For scaling laws: do you want the compute-optimal allocation result (how to split a fixed compute budget between model size and data), or the raw loss-vs-resource power laws? - For clustering: should I assume the number of clusters is known, and is the goal a hard partition or a probabilistic / generative description of the data? ### Part 1 — Optimizers Compare the main first-order optimizers used in deep learning: plain SGD, SGD with momentum, RMSProp, and Adam / AdamW. Explain what each adds, why adaptive methods like Adam are the default for training transformers, the difference between L2 regularization and **decoupled weight decay** (AdamW), and how the learning-rate schedule (warmup + decay) fits in. ```hint What each term buys you Trace the progression: gradient → add a velocity/momentum term → add per-parameter adaptive scaling (divide by a running RMS of gradients) → Adam combines both with bias correction. ``` ```hint The AdamW subtlety With Adam, adding the L2 penalty to the gradient gets divided by the adaptive denominator, so it is *not* equivalent to true weight decay; AdamW applies the decay directly to the weights instead. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2 — Scaling laws Explain neural scaling laws for language models. State the power-law form, what the Chinchilla (compute-optimal) result says about how to allocate a fixed compute budget between **model size** and **training tokens**, and the practical implications for choosing a model size given a compute or inference budget. ```hint The compute constraint For dense transformers, training compute is roughly $C \approx 6 N D$ (N parameters, D tokens); compute-optimal training means minimizing loss subject to that fixed $C$. ``` ```hint The headline finding Earlier work undertrained large models; the compute-optimal recipe scales N and D in roughly equal proportion — a smaller model trained on more tokens can beat a larger, data-starved one. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3 — K-means and Gaussian Mixture Models Compare **k-means** clustering and **Gaussian Mixture Models (GMM)** fit by Expectation-Maximization. Explain the objective each optimizes, hard vs. soft assignment, the precise relationship between them, and when you would choose one over the other. ```hint Same skeleton, different assignment Both alternate "assign points to clusters" and "update cluster parameters." K-means makes a *hard* assignment to the nearest centroid; GMM makes a *soft* (probabilistic) assignment via the E-step responsibilities. ``` ```hint K-means as a limit of GMM K-means is (essentially) the limiting case of a GMM with isotropic, equal, shared covariance $\sigma^2 I$ as $\sigma^2 \to 0$ — that limit forces responsibilities to 0/1 and the M-step reduces to the centroid mean. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions - Why does Adam sometimes generalize worse than well-tuned SGD with momentum, and when does that matter? - Given a fixed compute budget but a strict inference-latency target, how does that change the compute-optimal model size you would actually ship? - How would you choose the number of clusters $k$ for k-means vs. the number of components for a GMM, and what model-selection tools differ between them? - Your GMM's EM diverges with covariance entries collapsing toward zero. What is happening and how do you fix it?