Explain PPO and GRPO: Clipped Objective, Advantage Estimation and Group Baselines

Explain two policy-gradient methods used to train agents and language models with reinforcement learning: Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). Cover how each estimates advantages, what objective each optimizes, and when you would choose one over the other. The report says the candidate was asked basic reinforcement-learning questions on PPO, GRPO and other topics. It does not give the exact wording or name the other topics, so this version covers PPO, GRPO and how they compare. ### Clarifying Questions - Should the answer focus on fine-tuning language models, where an episode is a prompt and a sampled response, or on general control problems with multi-step environments? - Is the reward a learned reward model, or a verifiable check such as whether an answer is correct? - How much mathematical detail is expected: the full objectives, or the intuition behind them? ### Part 1 — PPO Explain what problem PPO solves compared with a plain policy-gradient update, write its objective, and walk through one training iteration, including how advantages are estimated. ```hint Reusing old samples Think about what goes wrong when you take many gradient steps on data collected by a policy that has since changed. ``` #### What This Part Should Cover - The probability ratio and the clipped surrogate objective, including why it takes a minimum - Advantage estimation with a learned value function, and the other terms in the loss - The collect-then-update loop and its key hyperparameters ### Part 2 — GRPO Explain how GRPO differs from PPO: where its baseline comes from, how advantages are computed and assigned to tokens, and how it keeps the policy close to a reference model. ```hint Where the baseline comes from PPO learns a critic to reduce variance. Ask what else could serve as a baseline when you sample several responses to the same prompt. ``` #### What This Part Should Cover - Group sampling and the group-normalized advantage - The objective, including clipping and the KL term - What removing the value model saves and what it costs ### Part 3 — Compare and choose When would you prefer PPO, and when GRPO? Describe the failure modes of each and what you would monitor during training. ```hint Count the models and the samples List the networks each method keeps in memory and how many samples each needs per prompt or state. ``` #### What This Part Should Cover - Compute, memory and sampling trade-offs - Credit assignment and the kinds of reward each method handles well - Failure modes and training diagnostics ### What a Strong Answer Covers - Correct objectives with every symbol explained - A clear account of why the methods need clipping and a baseline - Accurate differences between the two methods, without overstating either - Practical knowledge of failure modes such as reward hacking, groups that give no learning signal and effects of response length - A recommendation tied to the setting ### Follow-up Questions - If every response in a GRPO group receives the same reward, what update does that prompt produce, and what would you do about it? - In PPO's clipped objective, why take the minimum of the clipped and unclipped terms instead of simply clipping the ratio? - How does averaging the GRPO loss within each response, rather than over all tokens in the batch, affect long and short responses?