The attention mechanism

Self-attention combines token representations using weights computed from queries and keys. This lesson follows the calculation through projection, scoring, causal masking, softmax, and a weighted sum of values.

The previous lesson introduced token representations, positional information, and normalization. Attention now supplies a way for one position to use information from other positions in the prompt.

For example, processing "little" can depend on earlier tokens in "Twinkle, twinkle, little". The model needs a learned rule for how to combine that context, rather than a fixed assumption about which earlier word matters most.

Self-attention supplies that rule through numerical comparisons and weighted combinations.

The self-attention mechanism

Self-attention computes weights over the allowed positions in the same sequence and uses those weights to combine their value vectors.

The output at a position changes with the surrounding sequence. The token for “bank,” for example, may occur near words about a river or a financial account. Attention provides a way to combine contextual evidence; it does not assign a fixed human interpretation to each weight.