The learning objective and training loop

Follow a causal language model’s training step: predict tokens, calculate cross-entropy loss, backpropagate gradients, and update weights. A worked example shows how probabilities become a single loss value.

The inference examples used random weights to illustrate shapes and operations. Those weights have not learned the patterns needed to predict language.

Training repeatedly compares predictions with observed text and adjusts the weights. This lesson explains one training step. It is a conceptual walkthrough, not a complete training program.

Self-supervised learning

The pretraining objective considered here is causal language modeling: predict the next token from the tokens that precede it.

This objective is self-supervised because the text supplies its own targets. In the example “Twinkle, twinkle, little star,” the prefix can be used to predict the next observed token. We do not need a person to label each next-token pair. Data collection, filtering, and later training stages can still require human decisions and supervision.