Explain and sketch the Transformer architecture end to end

Quick Overview

Explain and sketch the Transformer architecture end to end, from token embeddings and positional encodings through encoder and decoder blocks to the output softmax. Tests scaled dot-product and multi-head attention, masking, cross-attention, normalization, tensor shapes, cost, and model variants.

Explain and sketch the Transformer architecture end to end

Company: Yahoo

Role: Machine Learning Engineer

Category: Machine Learning

Difficulty: easy

Interview Round: Onsite

In a machine learning fundamentals round, explain the architecture of the Transformer model and draw it on a shared whiteboard while you talk. Walk through how an input sequence is turned into output predictions, name every block in the diagram, and say what flows between the blocks. ```hint Follow one token Trace a single input token from its embedding to the output layer, and label the tensor shape on every arrow you draw. ``` ```hint Where Q, K and V come from For each attention block in your diagram, ask which sequence supplies the queries, which supplies the keys and values, and which positions each query is allowed to see. ``` ### Clarifying Questions - Should I describe the original encoder-decoder Transformer, a decoder-only model of the kind used for large language models, or both? - How deep should I go into the attention math compared with the overall block diagram? - Should the explanation cover how training differs from inference, for example teacher forcing versus step-by-step generation? ### What a Strong Answer Covers - The end-to-end data flow: token embeddings, positional information, stacked encoder and decoder blocks, output projection and softmax - Scaled dot-product attention with its formula and the reason for the scaling, and how multi-head attention splits and recombines the heads - The differences between encoder self-attention, masked decoder self-attention, and cross-attention - Residual connections, layer normalization and where it is placed, and the position-wise feed-forward network - Tensor shapes along the diagram, and how the cost of attention grows with sequence length - How encoder-only, decoder-only and encoder-decoder variants differ and what each is used for ### Follow-up Questions - Attention cost grows quadratically with sequence length. Where does that come from, and what are the main ways to reduce it? - How does a key-value cache speed up generation, and what does it cost in memory? - Why have most recent large models moved to pre-norm blocks and rotary or relative position encodings? - How would you adapt a pretrained Transformer to a text classification task?

Overview: Explain and sketch the Transformer architecture end to end, from token embeddings and positional encodings through encoder and decoder blocks to the output softmax. Tests scaled dot-product and multi-head attention, masking, cross-attention, normalization, tensor shapes, cost, and model variants.

Read the full Yahoo Machine Learning Engineer interview experience this question came from

Yahoo logo
Yahoo
Nov 8, 2026
easyMachine Learning EngineerOnsiteMachine Learning
1
0

In a machine learning fundamentals round, explain the architecture of the Transformer model and draw it on a shared whiteboard while you talk. Walk through how an input sequence is turned into output predictions, name every block in the diagram, and say what flows between the blocks.

Clarifying Questions Guidance

  • Should I describe the original encoder-decoder Transformer, a decoder-only model of the kind used for large language models, or both?
  • How deep should I go into the attention math compared with the overall block diagram?
  • Should the explanation cover how training differs from inference, for example teacher forcing versus step-by-step generation?

What a Strong Answer Covers Guidance

  • The end-to-end data flow: token embeddings, positional information, stacked encoder and decoder blocks, output projection and softmax
  • Scaled dot-product attention with its formula and the reason for the scaling, and how multi-head attention splits and recombines the heads
  • The differences between encoder self-attention, masked decoder self-attention, and cross-attention
  • Residual connections, layer normalization and where it is placed, and the position-wise feed-forward network
  • Tensor shapes along the diagram, and how the cost of attention grows with sequence length
  • How encoder-only, decoder-only and encoder-decoder variants differ and what each is used for

Follow-up Questions Guidance

  • Attention cost grows quadratically with sequence length. Where does that come from, and what are the main ways to reduce it?
  • How does a key-value cache speed up generation, and what does it cost in memory?
  • Why have most recent large models moved to pre-norm blocks and rotary or relative position encodings?
  • How would you adapt a pretrained Transformer to a text classification task?