From words to meaning: tokenization and embeddings

A language model needs a numerical representation of its input. Tokenization divides text into vocabulary items and assigns their IDs. An embedding lookup then supplies a vector for each ID. This lesson follows both steps and explains what information each representation carries.

The example prompt is "Twinkle, twinkle, little". Before the model can process it, the text must become a sequence of numbers.

The conversion has two stages. Tokenization identifies the pieces of text; embeddings supply the numerical vectors that the model will process. Keeping these stages separate avoids confusing a token’s vocabulary ID with its learned representation.

Breaking text into pieces with tokenization

When an LLM is given text as a string of characters, it first converts the string into a sequence of chunks. Tokenization is the process of breaking a piece of text into smaller units called tokens. These tokens can be words, parts of words (subwords), or even individual characters and punctuation.

Byte pair encoding

A tokenizer needs a rule for choosing text pieces. Byte pair encoding (BPE), used by the tokenizer in this lesson, builds a vocabulary by repeatedly merging common adjacent pairs.