Which Regularization Penalties Set Model Coefficients Exactly to Zero

Which regularization methods can shrink some of a model's coefficients exactly to zero, rather than only making them small? For a linear model trained with a regularized loss, go through the common penalties, at least L1 (lasso), L2 (ridge) and elastic net, and say for each whether it can produce exact zeros and why. ```hint Look near zero Compare how each penalty behaves as a coefficient approaches zero: does it keep pushing with the same force, or does the push fade away? ``` ### Clarifying Questions - Is the model solved to its exact optimum (for example linear or logistic regression with a dedicated solver), or trained with stochastic gradient descent, where exact zeros also depend on the optimizer? - Should techniques that are also called regularization, such as dropout or early stopping, be classified too? ### What a Strong Answer Covers - A correct classification of which penalties produce exact zeros and which only shrink - A mechanism: the penalty's slope at zero, or the geometry of the constraint region - A worked example of the regularized solutions on the same data - Practical consequences: feature selection, correlated features, feature scaling, and how the optimizer affects exact zeros ### Follow-up Questions - With two strongly correlated features, how do lasso, ridge and elastic net share weight between them? - Training with stochastic gradient descent and an L1 penalty rarely gives exact zeros. Why, and what fixes it? - What is the Bayesian interpretation of the L1 and L2 penalties? - How does the regularization strength control how many coefficients are zero, and how would you choose it?