Compare OLS, Ridge, and Lasso under Collinear Predictors

Explain how linear dependence among predictors affects ordinary least squares, ridge regression, and lasso. Discuss both fitted coefficients and statistical quantities used to interpret the fit. ### Constraints and Clarifications Use a linear regression model with an intercept and finite observations. Center predictors and the target for the algebra, leaving the intercept unpenalized. Compare standardized predictors when discussing coefficient penalties. The exact numerical datasets and statistical metrics were not specified; use the following clearly stated comparison cases: full column rank, near collinearity, and exact linear dependence. Specify which uncertainty or fit measures you discuss rather than assuming every reported metric behaves identically. ### Part 1 — Ordinary Least Squares Describe coefficient identification, prediction behavior, and coefficient uncertainty in the three cases. What happens if two predictor columns are identical? #### What This Part Should Cover Distinguish uniqueness of coefficients from uniqueness of fitted values. Explain when classical standard-error formulas apply. ### Part 2 — Ridge and Lasso Compare how positive regularization affects the objective, coefficient uniqueness, shrinkage, and feature selection under correlation. Explain the difference between a stable prediction and a stable selected feature set. #### What This Part Should Cover State the penalty conventions and avoid claiming that lasso always picks a unique representative of correlated predictors. ### What a Strong Answer Covers Correct algebra for OLS and ridge, the geometry of lasso's penalty, explicit assumptions behind statistical inference, and a practical plan to compare prediction and coefficient stability. ### Follow-up Questions - Why should predictor scaling be controlled before comparing penalties? - Can adding an exactly duplicated predictor change OLS fitted values while leaving the observations and original columns unchanged?