From raw text to helpful assistant

Learn how supervised fine-tuning and reinforcement learning from human feedback can adapt a base language model for assistant behavior. Distinguish demonstrations, preference labels, reward models, and policy updates.

Pretraining produces a model trained to predict text. That can support useful capabilities, but it does not directly specify how the model should respond to a user’s request.

Post-training aims to improve instruction following and other desired behaviors. Here we examine supervised fine-tuning and a reward-model-based RLHF pipeline. These are methods used in alignment, not a complete solution to bias, harmful output, or conflicting human preferences.

Stage 1: instruction fine-tuning (SFT)

A base model may treat “Explain the concept of black holes” as text to continue rather than a request to answer. Supervised examples can establish the desired pattern: a user asks a question, and the assistant gives a relevant response.

Supervised fine-tuning (SFT), often called instruction tuning in this context, trains on examples of the desired responses. The dataset can contain instruction-response pairs or conversations. Its size and how the examples are created depend on the project; quality must be judged against the intended behavior.