Implement Linear Regression with Batch Gradient Descent from Scratch

Implement linear regression trained with batch gradient descent, from scratch, in plain Python or NumPy and without a machine learning library. The assessment supplied the formulas; the version below uses the mean squared error convention, and every training step uses all examples. For $n$ examples with feature vectors $\mathbf{x}_i$ of length $d$ and targets $y_i$: - Model: $\hat{y}_i = \mathbf{w}^\top \mathbf{x}_i + b$. - Loss: $J(\mathbf{w}, b) = \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)^2$. - Gradients: $\frac{\partial J}{\partial \mathbf{w}} = \frac{2}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)\,\mathbf{x}_i$ and $\frac{\partial J}{\partial b} = \frac{2}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)$. - Update with learning rate $\alpha$: $\mathbf{w} \leftarrow \mathbf{w} - \alpha\frac{\partial J}{\partial \mathbf{w}}$ and $b \leftarrow b - \alpha\frac{\partial J}{\partial b}$. Start from $\mathbf{w} = \mathbf{0}$ and $b = 0$, and implement: ```python def fit_linear_regression( X: list[list[float]], # n examples, d features each y: list[float], # n targets learning_rate: float, num_iterations: int, ) -> tuple[list[float], float, list[float]]: """Return (w, b, losses), where losses[k] is J just before update k.""" def predict(X: list[list[float]], w: list[float], b: float) -> list[float]: ... ``` A check for your implementation: with `X = [[1.0], [2.0], [3.0]]`, `y = [3.0, 5.0, 7.0]` and `learning_rate = 0.1`, the parameters after one update are `w = [2.2667]`, `b = 1.0000`, and after two updates `w = [2.0178]`, `b = 0.8933` (rounded to four decimals). ```hint One residual vector Compute the predictions and residuals once per iteration and derive both gradients from them before changing either parameter. ``` ```hint Watch the loss Record the loss as you train; its trend tells you whether the learning rate is too small, about right, or too large. ``` ### Clarifying Questions - Should training also stop early when the loss stops improving, or always run the given number of iterations? - May I use NumPy, and should features be standardized inside the function or by the caller? - What should happen if the loss becomes infinite or NaN because the learning rate is too large? ### What a Strong Answer Covers - A correct implementation of the given formulas, with simultaneous updates of the weights and the bias - Vectorized (or clearly looped) code with the right array shapes, and its cost per iteration - The correct numbers on the small check - Learning-rate behaviour: slow convergence, divergence, and the effect of feature scaling - Edge cases: empty input, mismatched lengths, constant features and non-finite losses ### Follow-up Questions - On the check data, what happens with a learning rate of 0.2 instead of 0.1, and why? - How would you add L2 regularization to the loss and to the update? - When would you use the closed-form normal-equation solution instead, and when not? - How does the code change for mini-batch stochastic gradient descent?