Pretraining at scale

Examine the data, computation, and training controls involved in pretraining. Then distinguish the resulting base model’s text-prediction ability from the behavior expected of an assistant.

The previous lesson examined one training step: compute predictions, measure loss, calculate gradients, and update weights.

Pretraining repeats that process over a large corpus. The resulting behavior depends on the model, the data, and how the available computation is used. Scale matters, but a training step does not guarantee a measurable improvement on every task.

Creating a base model

In the causal language-model setup studied here, pretraining adjusts a model’s weights to predict tokens in a broad text corpus. The objective supplies many examples without requiring a separately authored answer for each one. Useful language capabilities can develop through this training, even though the immediate target is the next token.