Explain Layer Normalization in Transformers

Quick Overview

This question evaluates a candidate's understanding of normalization techniques in deep learning—specifically Layer Normalization in Transformer blocks—covering the LayerNorm equation and the roles of gamma, beta, and epsilon, as well as contrasts with BatchNorm and RMSNorm, in the Machine Learning domain.

Explain Layer Normalization in Transformers

Company: Amazon

Role: Applied Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Onsite

Explain Layer Normalization in Transformers. Where is it applied (pre-norm vs post-norm), and why does that choice affect gradient flow and training stability? Write the LayerNorm equation, discuss the roles of gamma/beta and epsilon, and contrast LayerNorm with BatchNorm and RMSNorm. Include guidance on initialization, placed-before vs placed-after residual connections, and implications for inference latency and memory.

Overview: This question evaluates a candidate's understanding of normalization techniques in deep learning—specifically Layer Normalization in Transformer blocks—covering the LayerNorm equation and the roles of gamma, beta, and epsilon, as well as contrasts with BatchNorm and RMSNorm, in the Machine Learning domain.

|Home/Machine Learning/Amazon
Amazon logo
Amazon
Sep 6, 2025
mediumApplied ScientistOnsiteMachine Learning
14
0

Layer Normalization in Transformers: Placement, Gradients, and Practical Trade-offs

Task

Explain Layer Normalization (LayerNorm) as used in Transformer blocks. Address:

  1. Where LayerNorm is applied: pre-norm vs post-norm, with precise formulas and where it sits relative to the residual connection.
  2. Why this placement affects gradient flow and training stability.
  3. The LayerNorm equation and the roles of gamma, beta, and epsilon.
  4. A contrast with BatchNorm and RMSNorm (what is normalized, dependence on batch, pros/cons).
  5. Practical guidance: initialization, placement choice (before vs after residual), and implications for inference latency and memory.

Assume a standard Transformer block contains two sublayers per block: Multi-Head Attention and a Feed-Forward Network (MLP), each wrapped by a residual connection.

Loading comments...

Browse More Questions

More Machine Learning•More Amazon•More Applied Scientist•Amazon Applied Scientist•Amazon Machine Learning•Applied Scientist Machine Learning