How LLMs Learn (The Training Loop)

A training-time estimate describes how much work a model and dataset require. The training loop explains what that work does: compute predictions, measure a loss, calculate gradients, and update parameters. Training duration depends on the configuration and measured throughput; it is not always hundreds of days.

This lesson focuses on next-token pretraining for a causal language model. Other language-model objectives exist, and instruction tuning or preference training adds other stages. The loop below is a useful starting point for understanding where training compute goes.

The prediction task

A token is a unit produced by a tokenizer. It may represent a word, a word fragment, punctuation, or whitespace. Given a sequence, a causal language model predicts the distribution of the next token using only the preceding context.

For an illustrative text example, the prefix is:

“The capital of France is”

The continuation in the example is:

“Paris”

The displayed words are not a claim about a particular tokenizer: “Paris” may occupy one or more tokens depending on the tokenizer and context. During training, the target is the continuation found in the dataset. It is not necessarily a verified statement about the world.