# Compare Language-Model Post-Training Methods Compare major language-model post-training methods, explaining which signal each uses, which model stage or problem it addresses, and the trade-offs in data, stability, control, and evaluation. ### Constraints & Assumptions - Separate supervised adaptation from preference or reward optimization. - Do not treat every method as interchangeable under a new name. - Include the reference policy, reward model, or preference data when a method requires one. ### Clarifying Questions to Ask - Is the goal task adaptation, instruction following, style control, or behavior alignment? - Are scalar rewards available, only pairwise preferences, or demonstrations? - Can fresh on-policy samples be generated? ```hint Map signal to objective For each method, name the training data, optimized loss, and failure mode it is intended to address. ``` ### What a Strong Answer Covers - Supervised fine-tuning and its role. - Reward-model plus policy-optimization approaches and direct preference objectives. - On-policy versus offline data, KL control, and reward-model dependence. - Evaluation, reward hacking, distribution shift, and method-selection criteria. ### Follow-up Questions 1. When can preference optimization reduce capability? 2. How would you choose between collecting better demonstrations and optimizing harder against preferences?