Compare Reward-Model RLHF and Direct Preference Optimization

Compare a typical reward-model-based RLHF pipeline with Direct Preference Optimization. Explain the data, training stages, objectives, and role of a reference policy in each approach. ### Constraints & Assumptions Use supervised fine-tuning followed by preference learning as the common setting. RLHF is a broad family; describe a representative reward-model plus policy-optimization pipeline rather than implying there is only one implementation. ### Clarifying Questions What form do preferences take? Is there an existing supervised model? Are responses collected online or fixed in a dataset? What behavior and evaluation criteria should improve? ### What a Strong Answer Covers Trace preference pairs through reward modeling or direct policy optimization, explain regularization toward a reference, and discuss data quality and evaluation beyond the training objective. ### Follow-up Questions Does DPO require a separately trained reward model? Does it eliminate human preference data? How might preference bias, distribution shift, or reward exploitation affect the final model?