Compare Batch Normalization and Layer Normalization

Compare batch normalization and layer normalization. Explain what each normalizes, how training and inference differ, and how you would diagnose a normalization-related training or serving problem. ### Constraints & Assumptions - State the activation shape and normalization axes; the names alone do not specify an implementation's layout. - Use a minibatch of feature vectors as the baseline, then explain how convolutional or sequence inputs change the axes. - Distinguish trainable scale/shift parameters from statistics estimated from data. ### Clarifying Questions to Ask - Is the model processing feature vectors, convolutional activations, or variable-length sequences? - What batch size is available during training and serving? - Are training and inference modes and saved normalization state handled consistently? ```hint Write down the axes Ask whether one example's normalized value depends on the other examples in the current batch. ``` ### What a Strong Answer Covers - The normalization equation and the mean/variance axes for each method. - Batch normalization's training statistics and typical running-statistics inference behavior. - Layer normalization's per-example statistics and behavior with small batches. - Learned affine parameters, numerical stabilization, and configuration-dependent details. - A debugging plan for batch-size changes or training/serving discrepancies. ### Follow-up Questions - Why can changing batch size affect batch normalization more directly than layer normalization? - What would you inspect if evaluation quality dropped when a trained model was switched into inference mode?