How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Onsite rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at Amazon during technical interviews.

Explain Layer Normalization in Transformers

Quick Overview

This question evaluates a candidate's understanding of normalization techniques in deep learning—specifically Layer Normalization in Transformer blocks—covering the LayerNorm equation and the roles of gamma, beta, and epsilon, as well as contrasts with BatchNorm and RMSNorm, in the Machine Learning domain.

Layer Normalization in Transformers: Placement, Gradients, and Practical Trade-offs

Task

Explain Layer Normalization (LayerNorm) as used in Transformer blocks. Address:

Where LayerNorm is applied: pre-norm vs post-norm, with precise formulas and where it sits relative to the residual connection.
Why this placement affects gradient flow and training stability.
The LayerNorm equation and the roles of gamma, beta, and epsilon.
A contrast with BatchNorm and RMSNorm (what is normalized, dependence on batch, pros/cons).
Practical guidance: initialization, placement choice (before vs after residual), and implications for inference latency and memory.

Assume a standard Transformer block contains two sublayers per block: Multi-Head Attention and a Feed-Forward Network (MLP), each wrapped by a residual connection.

Layer Normalization in Transformers: Placement, Gradients, and Practical Trade-offs

Task

Explain Layer Normalization (LayerNorm) as used in Transformer blocks. Address:

Where LayerNorm is applied: pre-norm vs post-norm, with precise formulas and where it sits relative to the residual connection.
Why this placement affects gradient flow and training stability.
The LayerNorm equation and the roles of gamma, beta, and epsilon.
A contrast with BatchNorm and RMSNorm (what is normalized, dependence on batch, pros/cons).
Practical guidance: initialization, placement choice (before vs after residual), and implications for inference latency and memory.

Assume a standard Transformer block contains two sublayers per block: Multi-Head Attention and a Feed-Forward Network (MLP), each wrapped by a residual connection.

Explain Layer Normalization in Transformers

Quick Overview

Explain Layer Normalization in Transformers

Layer Normalization in Transformers: Placement, Gradients, and Practical Trade-offs

Task

Write your answer

Explain Layer Normalization in Transformers

Quick Overview

Explain Layer Normalization in Transformers

Layer Normalization in Transformers: Placement, Gradients, and Practical Trade-offs

Task

Write your answer