Positional encodings and layer normalization

Token embeddings identify the text pieces in a sequence. Positional encodings add information about where those pieces occur, while layer normalization controls the scale of a token’s activations. This lesson examines the two operations separately before attention combines information across tokens.

The previous lesson produced a matrix with shape (number_of_tokens, embedding_dimension). Each row contains the initial vector for one token occurrence.

Self-attention can process the prompt’s token representations in parallel, but the attention calculation needs a positional signal to make use of their order. Merely presenting token vectors in rows does not give that calculation the same sequential structure as a recurrent model.

The distinction matters for sentences such as "dog chases cat" and "cat chases dog". They contain the same words, but their order changes who performs the action.