Compare RMSNorm and LayerNorm in Transformer Blocks

Compare LayerNorm and RMSNorm in a transformer block. Write their equations, explain the effect of omitting mean subtraction, and discuss why RMSNorm can be an attractive design choice. ### Constraints & Assumptions Normalize over the hidden features of one token. Use the common affine LayerNorm and scale-only RMSNorm definitions; implementations can vary in bias, epsilon placement, and precision. ### Clarifying Questions Which axes are normalized? Is a learned bias included? Is normalization before or after the sublayer? What numerical precision is used for reductions? ### What a Strong Answer Covers Distinguish variance from mean-square normalization, identify invariances and computational differences, and avoid claiming that one normalization always produces better models. ### Follow-up Questions What happens if the same constant is added to every feature? How do epsilon and low-precision arithmetic affect the result? Does RMSNorm guarantee a zero-mean output?