Contrast L1 and L2 regularization effects
Overview
This question evaluates a candidate's understanding of regularization techniques in supervised learning, covering the geometric intuition behind L1 versus L2 penalties, sparsity and bias–variance trade-offs, behavior under multicollinearity and elastic net considerations, algorithmic solution differences, and the penalized logistic regression objective and tuning. It is commonly asked in the Machine Learning domain to assess both conceptual understanding and practical application of model regularization, numerical stability, and reproducible hyperparameter selection, and the expected level spans conceptual reasoning and implementation-aware details.
Ridge (L2) vs Lasso (L1) in Linear and Logistic Regression
Context: You are comparing L2 (Ridge) and L1 (Lasso) regularization for linear and logistic regression. Assume predictors are columns of X, response is y, and the intercept is not penalized unless stated. We write optimization in penalized form (minimize loss + penalty).
(a) Shrinkage geometry and sparsity
- Explain the geometric intuition of L1 vs L2 constraints/penalties and why L1 tends to yield exact zeros (sparse solutions), while L2 rarely does.
(b) Bias of penalized estimates
- Are L1/L2 penalized estimators unbiased? Provide a derivation or counterexample and discuss the bias–variance trade-off.
(c) Multicollinearity
- Under strong multicollinearity, which penalty is preferable and why? Briefly mention when elastic net might be better.
(d) Algorithms and library differences
- Describe coordinate descent and solution path algorithms (e.g., LARS). Explain why solutions can differ across libraries due to: penalty scaling choices, standardization/centering, intercept treatment, and λ-grid construction and tolerance.
(e) Logistic regression with L2
- Write the penalized (negative) log-likelihood to minimize and its gradient for L2-penalized logistic regression (intercept unpenalized). Describe how you would select λ using nested cross-validation, including key guardrails to avoid leakage.
Solution
Login required