Walk through the mathematical foundations that connect logistic regression to modern deep-learning training. The interviewer expects you to write the equations, derive the gradients (not just state them), and explain the design choices behind the optimizer — this is a whiteboard-style "show your math" round, not a conceptual chat. ### Constraints & Assumptions - Binary classification with input $x \in \mathbb{R}^d$ and label $y \in \{0, 1\}$. - You should be able to derive results from first principles; quoting a final formula without the chain-rule steps will not pass. - Use vectorized/matrix notation where natural (e.g. a batch design matrix $X \in \mathbb{R}^{N \times d}$). - Standard definitions apply: $\sigma(\cdot)$ is the logistic sigmoid, log is natural log, $\eta$ (or $\alpha$) is a learning rate. ### Clarifying Questions to Ask - Should derivations be from first principles, or is stating the final gradient (with a one-line justification) acceptable? - Do you want scalar (single-example) notation, or fully vectorized matrix notation over a batch? - For the neural network, should I assume a specific activation (sigmoid/ReLU) for hidden layers, or keep $g^{[l]}$ general? - For Part 1's gradient, do you want the single-example form, the batch-averaged form, or both? - For Adam, are you interested in the bias-correction terms and the role of $\epsilon$, or just the core moment estimates? ### Part 1 — Logistic regression - Explain how logistic regression models binary classification. - Write the model (linear score $\to$ sigmoid), the sigmoid function itself, and the binary cross-entropy (BCE) loss for one example and for a batch of $N$ examples. - Derive the gradient of the loss with respect to the parameters $w$ and $b$, and state the gradient-descent update. ```hint Where to start Write $z = w^\top x + b$, $p = \sigma(z)$, then differentiate the BCE loss. Keep $\dfrac{\partial L}{\partial z}$ as an intermediate — the chain rule factors as $\dfrac{\partial L}{\partial w} = \dfrac{\partial L}{\partial z}\cdot \dfrac{\partial z}{\partial w}$. ``` ```hint The key simplification The sigmoid has the convenient derivative $\sigma'(z) = \sigma(z)\,(1-\sigma(z))$. When you combine it with the BCE loss, watch for terms that cancel — the messy $p(1-p)$-style factors are exactly what makes the result collapse into one clean expression. Aim for that cancellation. ``` #### What This Part Should Cover - Probabilistic framing: $p = \sigma(w^\top x + b)$ read as $P(y=1\mid x)$, and BCE identified as the Bernoulli negative log-likelihood (not an arbitrary loss choice). - The chain rule carried *through* the sigmoid-derivative cancellation to reach the compact $\partial L/\partial z = p - y$ — shown, not quoted. - Both the single-example gradient and the vectorized batch gradient $\tfrac{1}{N}X^\top(p-y)$, plus the gradient-descent update and a note that the objective is convex in $(w,b)$. ### Part 2 — From logistic regression to a neural network - Show that logistic regression is exactly a one-layer ("no hidden layer") neural network, and say what changes when you add hidden layers. - For a feedforward network with $L$ layers, write the forward pass mathematically, then derive the backpropagation equations for the weights and biases. ```hint Forward pass Per layer $l$: $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, $a^{[l]} = g^{[l]}(z^{[l]})$, with $a^{[0]} = x$. Define a per-layer "error" $\delta^{[l]} = \partial L / \partial z^{[l]}$ to organize the recursion. ``` ```hint The backward recursion The whole derivation hangs on writing $\delta^{[l]}$ in terms of $\delta^{[l+1]}$ — set up that recurrence and the per-weight gradient falls out of the chain rule. As you do, keep track of which factor moves the error to the previous layer and which factor accounts for the local activation. Let the matrix shapes guide where each term goes. ``` #### What This Part Should Cover - A correct statement that logistic regression is the $L=1$ (no-hidden-layer) case, and a clear account of what hidden layers add (learned non-linear features before the final linear-plus-sigmoid). - A correct forward pass and a backprop recursion expressed via the per-layer error $\delta^{[l]}$, including the elementwise activation-derivative factor. - Weight/bias gradients with correct shapes (outer product per example, matrix product for a batch), and why backprop is $O(\text{forward cost})$ via cached activations. ### Part 3 — Mini-batch gradient descent - Write clear pseudocode for training a model with mini-batch gradient descent, and note why mini-batches are preferred over full-batch or pure SGD. ```hint Structure Two nested loops: an outer epoch loop (shuffle each epoch) and an inner loop over batches of size $B$. Each batch does forward $\to$ loss $\to$ backward $\to$ parameter update. Think about where averaging over the batch happens. ``` #### What This Part Should Cover - Pseudocode that shuffles each epoch, partitions into batches of size $B$, and runs forward $\to$ batch-averaged loss $\to$ backward $\to$ update — with the $1/B$ averaging placed so the gradient scale is roughly batch-size-independent. - The tradeoff that motivates mini-batches: hardware throughput and lower gradient variance vs. full-batch, while retaining enough stochasticity (and bounded memory) vs. pure SGD. ### Part 4 — Adam optimizer - Describe Adam's main components and write its update rule (first moment, second moment, bias correction, parameter step). - Explain *why* Adam scales each update by the square root of the second moment of the gradient. ```hint Two ideas combined Adam fuses momentum (an EMA of the gradient $g_t$) with an adaptive per-parameter step size (an EMA of the *squared* gradient $g_t^2$). Write both EMAs, then reason about why the moving averages need bias correction early in training. ``` ```hint Why the square root Look at the units. The second moment $v_t$ accumulates $g^2$, so ask: what are its units, and what happens to the units of the step $\hat m_t / \sqrt{\hat v_t}$ once you take the root? Reason from there about what that does to the per-parameter step magnitude. ``` #### What This Part Should Cover - All four ingredients written correctly: first-moment EMA, second-moment EMA, bias correction (and *why* it's needed when $m_0=v_0=0$), and the parameter update with $\epsilon$. - A real argument — not just restatement — for the $\sqrt{\hat v_t}$ denominator: dimensional consistency / scale-invariance of the step, and per-parameter adaptive step sizing that protects rare or large-gradient directions. ### What a Strong Answer Covers These dimensions span all four parts; the per-Part rubrics above cover the part-specific content. - **Notation discipline**: one consistent notation reused across parts (e.g. $\delta^{[L]} = a^{[L]} - y$ reusing the Part 1 cancellation), so the parts visibly connect rather than reading as four disconnected facts. - **Derive, don't quote**: every gradient is reached by visible chain-rule steps; final formulas are the endpoint of a derivation, not the starting point. - **Intuition alongside algebra**: each result is accompanied by a one-line "what this means" (residual-times-input gradient, error propagated backward, momentum + adaptive scaling), and stated assumptions are surfaced up front. ### Follow-up Questions - Why is the BCE loss preferred over squared error for classification? What happens to the gradient/optimization landscape if you use MSE with a sigmoid output? - During backprop with sigmoid/tanh activations, what is the vanishing-gradient problem and how do ReLU, normalization, or residual connections mitigate it? - How does Adam differ from RMSProp and from SGD with momentum? When might plain SGD (with a tuned schedule) generalize better than Adam? - What is weight decay, and why is decoupled weight decay (AdamW) different from adding L2 regularization to the loss when using Adam?