Explain transformer architecture and FFN rationale
Overview
This question evaluates a candidate's understanding of Transformer architecture components and competencies in attention mechanics, the rationale for position-wise feed-forward networks (FFNs), tensor shape reasoning for multi-head attention, computational complexity analysis, and implementation trade-offs such as projection strategies and positional encodings. It is commonly asked to assess architectural reasoning and practical scalability considerations in deep learning sequence models, falls under the Machine Learning domain (deep learning/sequence modeling), and requires both conceptual understanding and practical application-level knowledge.
Transformer Architecture: Components, Shapes, Complexity, and Design Choices
Task
Explain the Transformer architecture in detail for a standard encoder–decoder model used in sequence modeling.
-
Layer composition
- What sublayers are in each encoder and decoder layer?
- Why include a position-wise feed-forward network (FFN)?
- How do residual connections and layer normalization interact (pre-LN vs post-LN)?
-
Multi-head attention shapes
- For d_model = 512, h = 8 heads, and sequence length n, give the tensor shapes for Q, K, V, attention scores, attention weights, per-head outputs, the concatenated output, and the final projection output. State shapes both per-head and stacked across heads; you may omit the batch dimension for clarity.
-
Computational complexity
- Derive the time and memory complexity of self-attention with respect to n and d (assume d = d_model and typical d_k = d_v = d/h). Also note cross-attention complexity.
-
Implementation choices and trade-offs
- Q/K/V projections (separate vs fused, multi-query/grouped-query attention).
- Number of heads and head dimension.
- Positional encodings (sinusoidal, learned, relative/rotary/ALiBi).
- Include practical trade-offs (speed, memory, quality, length generalization).
Solution
Login required