You are in a phone screen for an applied scientist / machine-learning engineer role and are asked to verbally explain a set of machine-learning fundamentals. For each part, give a precise, conceptually correct answer and be ready to justify *why*, not just *what*. Treat each question as an invitation to demonstrate depth: state the core idea, then explain the reasoning or intuition behind it. ### Constraints & Assumptions - This is a conceptual / whiteboard-style discussion, not a coding exercise. No data, libraries, or runnable code are provided. - Answers are expected to be verbal explanations with light math notation where helpful (e.g. loss functions, update rules). - Assume standard supervised-learning settings unless a part specifies otherwise. - Depth and correctness of reasoning matter more than breadth; the interviewer probes the "why" behind each answer and will follow up on hand-wavy claims. ### Clarifying Questions to Ask - For the regression and classification parts, should I focus on the modeling assumptions, the estimation/optimization view, or both? - When discussing loss functions, do you want the probabilistic (maximum-likelihood) justification, or just the optimization properties? - For the optimizer comparison, are you interested in a specific regime (e.g. large-scale vision, NLP/transformers, sparse features), or a general comparison? - For the neural-network part, are we reasoning about classical small-network intuition or the modern overparameterized deep-learning view? - How much depth do you want per part — a one-paragraph summary each, or should I go deep on the one I find most interesting? ### Part 1 — Linear Regression What are the main assumptions of linear regression? Why is squared loss commonly used? ```hint Where to start List the classical assumptions one at a time (think: functional form of the model, the error term's conditional mean, independence / correlation of errors, error variance, and relationships among features). Then sort them into "needed for unbiased point estimates" vs. "needed for valid inference / standard errors." ``` ```hint Why squared loss Ask what probabilistic noise model makes least-squares the **maximum-likelihood** estimator. Also weigh convexity, differentiability, and which statistic of $y$ squared loss ends up estimating (the conditional mean vs., say, the median). ``` #### What This Part Should Cover - States the classical assumptions by name and correctly classifies each as required for **unbiasedness/consistency** of the point estimate vs. required only for **valid inference** (standard errors, CIs, hypothesis tests). - Identifies which assumption is *not* needed for unbiasedness and explains why it is sometimes added. - Gives at least two distinct justifications for squared loss (one probabilistic, one optimization-theoretic), and names which statistic of $y$ the minimizer targets. ### Part 2 — Logistic Regression What is logistic regression? Why do logarithms appear in its formulation or loss function? ```hint Where the log enters The log shows up in *two distinct places* once you write the model out. Trace the path from a raw probability in $(0,1)$ to an unconstrained linear score, and separately think about how the parameters are actually fit. Ask what each step would look like *without* a log and why that breaks. ``` ```hint The loss For Bernoulli labels, the likelihood is a *product* of per-example probabilities. What does taking a $\log$ do to a product, and why is that helpful both mathematically (turning the objective into a sum) and numerically? ``` #### What This Part Should Cover - Correctly states what logistic regression models and writes out (or describes) the relationship between the linear score and the output probability. - Identifies and explains **both** distinct places where a logarithm appears — one in the model form and one in the fitting criterion — with a clear reason why each is necessary or convenient. - Explains the practical benefit(s) of the log in the fitting criterion beyond just "it's the MLE." ### Part 3 — Random Forest What is a random forest? During tree construction, how is the set of candidate features at each split selected? ```hint Two sources of randomness A random forest injects randomness in two independent ways: how the *data* for each tree is drawn, and how *features* are considered at each split. Name both, and be ready to say which one the question is really probing. ``` ```hint Feature selection at a split Consider whether each split may look at *all* features or only a restricted random subset, and what tuning knob controls that count. Then push on *why* deliberately hiding features from a split can make the overall ensemble better rather than worse. ``` #### What This Part Should Cover - Gives a clear definition of the ensemble and how its prediction is formed (vote vs. average). - Names **both** sources of randomness accurately, including the relevant hyperparameter for the feature-selection mechanism and common default values. - Provides a principled explanation — grounded in ensemble theory, not just intuition — for *why* restricting features at each split improves the ensemble's generalization. ### Part 4 — Adam vs. SGD Explain the Adam optimizer. What are its advantages and disadvantages compared with vanilla stochastic gradient descent? ```hint What state Adam keeps Adam fuses two ideas you've seen elsewhere by keeping per-parameter running statistics of the gradient stream. What two quantities about recent gradients would each idea track, and how would the update combine them? Once named, write the moving-average updates, the bias-correction step, and the final parameter update. ``` ```hint Trade-offs to weigh Be honest about both sides: faster early convergence and per-parameter adaptive step sizes vs. extra memory (two states per parameter) and the documented generalization gap relative to well-tuned SGD with momentum. Mention how weight decay interacts with Adam (L2 vs. decoupled / AdamW). ``` #### What This Part Should Cover - Correctly names and describes the **two running statistics** Adam maintains per parameter and the conceptual ideas each comes from. - Explains the bias-correction step — what causes the bias and why correction is needed — and writes or describes the final update formula. - Provides at least two concrete **advantages** and at least two concrete **disadvantages** or caveats relative to SGD, including the weight-decay interaction. ### Part 5 — Narrow vs. Wide Networks and Local Minima Consider two neural networks with the same two-layer structure. One has only a few neurons per layer; the other has many neurons per layer. Which one is more likely to get trapped in a poor local minimum, and why? ```hint Frame it as capacity and landscape Both objectives are non-convex. Reason about how the number of parameters affects the *number of low-loss configurations* and how "connected" the good solutions are in the loss landscape (isolated bad basins vs. wide connected low-loss regions). ``` ```hint Don't forget the trade-off A complete answer names which network is more prone to poor local minima / underfitting — *and* flags the cost of the easier-to-optimize one: what does extra capacity risk when data or regularization is limited? ``` #### What This Part Should Cover - Makes a **clear, unambiguous choice** of which architecture is more susceptible, with a capacity-based justification. - Explains the loss-landscape argument for *why* the other architecture is easier to optimize, going beyond "more parameters = better." - Acknowledges the **countervailing risk** that comes with the easier-to-optimize network and names at least one practical mitigation. --- ### What a Strong Answer Covers The interviewer is listening for these cross-cutting signals across all five parts (this is a checklist of *dimensions* the interviewer scores, not the answers themselves): - **Assumptions stated explicitly** for linear and logistic regression, with awareness of which ones matter for unbiased point estimates vs. valid inference. - **Probabilistic grounding**: connecting squared loss and log-loss to maximum likelihood under specific noise / label models. - **Mechanism of randomness** in ensembles and *why* it helps (variance reduction via decorrelation), not just "it's a bunch of trees." - **Optimizer internals**: what state Adam maintains, the actual update rule, and honest trade-offs vs. SGD (memory, generalization, tuning, weight decay). - **Non-convex optimization intuition** for narrow vs. wide networks, including capacity, the structure of the loss landscape, and overfitting risk. - **Calibrated nuance**: acknowledging where the textbook answer is incomplete, or where practice diverges from theory. ### Follow-up Questions - For squared loss: how would your answer change if the noise were heavy-tailed (e.g. Laplacian) instead of Gaussian — what loss would maximum likelihood give you then, and what statistic of $y$ would it estimate? - For random forests: how do `n_estimators` and the feature-subset size $m_{try}$ trade off bias, variance, and decorrelation between trees? - For Adam: in what concrete settings have you seen (or would you expect) SGD with momentum to generalize better, and what would you try to close the gap? - For the narrow-vs-wide question: how does the modern overparameterization view (loss-landscape connectivity, flat vs. sharp minima) reconcile with the classical "more parameters → more overfitting" intuition?