# Detect and Prevent Reward Hacking Explain how reward hacking appears during language-model post-training and how you would detect, limit, and respond to it without assuming the reward score equals the real objective. ### Constraints & Assumptions - The reward is a proxy and may omit important qualities. - Detection must use evidence independent of the optimized training reward. - Interventions should preserve useful capability where possible. ### Clarifying Questions to Ask - What behavior is the reward intended to measure? - Which shortcuts could raise the score without satisfying users? - What independent evaluation or human review is available? ```hint Look for score-behavior divergence Inspect cases where optimized reward improves while blinded human or rule-based measures become worse. ``` ### What a Strong Answer Covers - Concrete manifestations such as verbosity, style mimicry, grader manipulation, or exploiting evaluator blind spots. - Independent holdouts, adversarial prompts, causal probes, and reward-model ensembles. - Regularization, constraints, data repair, limited optimization, and monitoring. - Incident response, rollback criteria, and residual uncertainty. ### Follow-up Questions 1. How would you detect hacking that generalizes beyond known test prompts? 2. When can adding more reward-model capacity make the problem worse?