Explain SFT, DPO, PPO, and the Role of KL Regularization

Explain the objectives and practical differences among supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and PPO-based language-model post-training. ### Part 1 — SFT and the Role of Demonstrations Explain the SFT training objective and why a workflow may still use SFT before preference optimization. #### What This Part Should Cover Conditional token likelihood on demonstrations, loss masking, and the distinction between learning demonstrated behavior and comparing preferred alternatives. ### Part 2 — DPO and PPO Explain what standard DPO optimizes and how its training workflow differs from PPO-based reinforcement learning with preference-derived rewards. Discuss why SFT and DPO may remain useful rather than assuming one method always replaces the others. #### What This Part Should Cover Chosen/rejected response pairs, the reference policy, relative log probabilities, and the roles of sampling, reward estimation, and policy updates. ### Part 3 — The KL Question Assess the statement that DPO does not need KL regularization. Distinguish the absence of a separately computed KL penalty in the usual DPO loss from the KL-regularized derivation and reference-policy dependence. #### What This Part Should Cover A precise correction of the premise and the role of the DPO temperature or regularization parameter. ### Constraints Discuss standard formulations rather than claiming that every later variant behaves identically. Use an autoregressive language model, demonstration data for SFT, and response-preference pairs for DPO. No task-specific winner is assumed. ### Clarifying Questions - Are good demonstrations, preference pairs, or an online reward signal available? - Is the question about the training loss, the full data-collection workflow, or deployment quality? ```hint Compare ratios relative to the reference DPO compares how the policy changes the odds of the preferred response relative to a fixed reference policy. ``` ### What a Strong Answer Covers - Correct SFT likelihood and DPO preference objectives. - A practical distinction between offline preference fitting and PPO-style rollout optimization. - A qualified explanation of KL and method selection based on data and evaluation. ### Follow-up Questions - What can happen when preference data poorly cover the model's deployed behavior? - Why does better training preference accuracy not guarantee better task performance?