Explain and sketch the Transformer architecture end to end
Company: Yahoo
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: easy
Interview Round: Onsite
In a machine learning fundamentals round, explain the architecture of the Transformer model and draw it on a shared whiteboard while you talk. Walk through how an input sequence is turned into output predictions, name every block in the diagram, and say what flows between the blocks.
```hint Follow one token
Trace a single input token from its embedding to the output layer, and label the tensor shape on every arrow you draw.
```
```hint Where Q, K and V come from
For each attention block in your diagram, ask which sequence supplies the queries, which supplies the keys and values, and which positions each query is allowed to see.
```
### Clarifying Questions
- Should I describe the original encoder-decoder Transformer, a decoder-only model of the kind used for large language models, or both?
- How deep should I go into the attention math compared with the overall block diagram?
- Should the explanation cover how training differs from inference, for example teacher forcing versus step-by-step generation?
### What a Strong Answer Covers
- The end-to-end data flow: token embeddings, positional information, stacked encoder and decoder blocks, output projection and softmax
- Scaled dot-product attention with its formula and the reason for the scaling, and how multi-head attention splits and recombines the heads
- The differences between encoder self-attention, masked decoder self-attention, and cross-attention
- Residual connections, layer normalization and where it is placed, and the position-wise feed-forward network
- Tensor shapes along the diagram, and how the cost of attention grows with sequence length
- How encoder-only, decoder-only and encoder-decoder variants differ and what each is used for
### Follow-up Questions
- Attention cost grows quadratically with sequence length. Where does that come from, and what are the main ways to reduce it?
- How does a key-value cache speed up generation, and what does it cost in memory?
- Why have most recent large models moved to pre-norm blocks and rotary or relative position encodings?
- How would you adapt a pretrained Transformer to a text classification task?
Overview: Explain and sketch the Transformer architecture end to end, from token embeddings and positional encodings through encoder and decoder blocks to the output softmax. Tests scaled dot-product and multi-head attention, masking, cross-attention, normalization, tensor shapes, cost, and model variants.
Read the full Yahoo Machine Learning Engineer interview experience this question came from