Explain Neural-Network Regularization and Dropout

Explain how regularization reduces overfitting in a neural network. Compare L1 and L2 penalties with dropout, including how dropout behaves during training and inference. ### Constraints & Assumptions - Explain the mechanism rather than only naming techniques. - Use standard inverted dropout as the example and state the drop probability convention. - Distinguish a penalty added to the objective from changes to architecture, data, or the optimization process. ### Clarifying Questions to Ask - Is the model overfitting, underfitting, or suffering from data leakage or distribution mismatch? - Which layers use dropout, and is the model in training or inference mode? - Does the optimizer implement an L2 gradient penalty or decoupled weight decay? ```hint Preserve the expected activation If a training activation is randomly set to zero, determine how surviving activations should be scaled so the conditional expected activation remains unchanged. ``` ### What a Strong Answer Covers - L1/L2 objective penalties and their different effects on parameters. - Bernoulli masking and inverted-dropout scaling. - The train/inference mode difference and why dropout is not simply deleting neurons permanently. - Data augmentation and early stopping as distinct regularization choices. - Diagnosis and validation rather than assuming more regularization always improves the model. ### Follow-up Questions - Why can too much dropout worsen both training and validation performance? - Why are an L2 penalty and decoupled weight decay not generally equivalent under adaptive optimizers?