The life of a prompt, end to end

Trace one next-token prediction from tokenization through the transformer stack to the output layer. Use this recap to connect the components from the preceding lessons.

The preceding lessons described a simplified decoder-only transformer. This page puts those components in order.

Follow the prompt "Twinkle, twinkle, little" through one prediction step. A trained model might continue it with “star”; the random-weight examples from earlier lessons do not establish that result.

Phase 1: input preparation

Input preparation converts the string into vectors and provides information about token positions:

1. Tokenization: Split the string according to the tokenizer’s vocabulary and assign each token its integer ID. Tokens can be words, word pieces, or punctuation.

2. Embedding: Look up each ID in the learned embedding table. Each lookup returns a vector, and the vectors form a matrix in sequence order.

3. Positional encoding and normalization: In our simplified example, add positional vectors to the embeddings and normalize the result. Position supplies order information; normalization controls feature scale. Other architectures supply position information or place normalization differently.