Implement and derive backprop from scratch

Q: Implement and derive backprop from scratch

This question evaluates understanding and practical implementation of neural network fundamentals, specifically analytic backpropagation and gradient derivation, numerically stable binary cross-entropy computation, parameter initialization, gradient descent updates, and gradient checking.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Onsite rounds at Anthropic.

Q: What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Anthropic during technical interviews.

Question

Tiny Neural Network From First Principles: Binary Classification

Implement and analyze a minimal neural network for binary classification with a single hidden layer, using vectorized NumPy (or a similar array library) without autograd — every gradient must be derived and coded by hand.

Assume a dataset with features $X \in \mathbb{R}^{N \times D}$ and labels $y \in \{0,1\}^N$ . The network is:

Hidden layer: $H$ units with an activation $f$ (ReLU or tanh).
Output layer: a single unit with a sigmoid producing $P(y=1 \mid x)$ .

The deliverable is a complete, self-contained training pipeline (forward, loss, backward, optimization, gradient check) plus a short written discussion of the numerical and design choices. The question is split into six parts below.

Constraints & Assumptions

Parameter shapes are fixed: $W_1 \in \mathbb{R}^{D \times H}$ , $b_1 \in \mathbb{R}^{H}$ , $W_2 \in \mathbb{R}^{H \times 1}$ , $b_2 \in \mathbb{R}^{1}$ .
Computation is vectorized over the batch — no Python loops over the $N$ examples in the forward/backward path.
No autograd / no deep-learning framework gradients — analytic derivatives only. Finite differences are allowed solely for the gradient-check verification step (Part 5).
Work in float64 for the implementation and especially for gradient checking.
The loss is the mean (not sum) binary cross-entropy over the batch.

Clarifying Questions to Ask

Should the loss be the mean or sum over the batch? (This decision changes the gradient scale and how the learning rate is interpreted.)
Which hidden activation is in scope — ReLU, tanh, or both ? (It changes the derivative term and the recommended initialization.)
Is mini-batch SGD expected, or is full-batch / single-batch gradient descent sufficient for the deliverable?
What floating-point precision should the reference target, and how tight should the gradient-check tolerance be?
Should the numerically-stable loss be expressed in terms of the probability $p$ or the logit $z_2$ ? (This is the crux of the stability design.)
Is a runnable end-to-end demo on a toy dataset expected, or just the component functions?

Part 1 — Forward pass

Compute, in vectorized form:

z_1 = X W_1 + b_1, \qquad a_1 = f(z_1), \qquad z_2 = a_1 W_2 + b_2, \qquad p = \sigma(z_2),

where $\sigma$ is the logistic sigmoid. State the shape of each intermediate and confirm the biases broadcast correctly across the $N$ rows.

What This Part Should Cover Premium

Part 2 — Loss (numerically stable binary cross-entropy)

Implement mean binary cross-entropy. The naive form $-[y\log p + (1-y)\log(1-p)]$ blows up once $p$ rounds to exactly $0$ or $1$ . Use a stable formulation (e.g. softplus $\log(1+e^{x})$ or log-sum-exp) so the loss never produces $\pm\infty$ or NaN.

What This Part Should Cover Premium

Part 3 — Backward pass (analytic gradients, no autograd)

Derive and implement $\partial L/\partial W_1$ , $\partial L/\partial b_1$ , $\partial L/\partial W_2$ , $\partial L/\partial b_2$ by the chain rule. State the derivative of your hidden activation explicitly.

What This Part Should Cover Premium

Part 4 — Optimization

Implement vanilla gradient-descent updates $\theta \leftarrow \theta - \eta\, g_\theta$ for all four parameters.

What This Part Should Cover Premium

Part 5 — Gradient checking

Verify your analytic gradients against central finite differences

g_{\text{num}} \approx \frac{L(\theta + \varepsilon) - L(\theta - \varepsilon)}{2\varepsilon},

and report the relative error per parameter.

What This Part Should Cover Premium

Part 6 — Discussion

Write short notes on each:

Numerical stability — sigmoid/logistic loss, softplus / log-sum-exp, log1p , expm1 , clipping.
Initialization — He vs Xavier, and bias initialization.
Activation choices — ReLU, tanh, sigmoid; pros and cons.
Batch size and gradient variance — and how it informs learning-rate scaling.

What This Part Should Cover Premium

What a Strong Answer Covers Premium

Follow-up Questions

Where exactly does the naive 1 - p lose precision, and which primitive ( expm1 / log1p ) recovers the tiny complement probability when $p$ saturates to $1$ ?
A hidden ReLU unit outputs $0$ for every training example. What has happened, how would you detect it, and how would you mitigate it?
How does this stable-loss construction generalize from binary (softplus on one logit) to the multiclass softmax cross-entropy case?
If you double the batch size, how should you adjust the learning rate, and what would you monitor to know whether the adjustment was too aggressive?

Implement and derive backprop from scratch

Quick Overview