Explain Transformer Encoder and Decoder Behavior
Company: Point72
Role: Quantitative Researcher
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
Answer the following Transformer fundamentals questions in a machine learning interview:
1. What are the main differences between a Transformer encoder and a Transformer decoder? Your answer should discuss attention patterns and the role of a causal mask.
2. For the same input prompt, why can an autoregressive decoder-based language model produce different outputs on different runs? Discuss sampling, temperature, and when generation would be deterministic.
Overview: This question evaluates knowledge of Transformer architecture and generative language modeling, specifically the encoder versus decoder roles, attention patterns and causal masking, plus mechanisms behind stochastic text generation such as sampling and temperature.