Compare GRPO and PPO and Explain Sparse Versus Dense Rewards
Company: Siemens
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
Compare Group Relative Policy Optimization (GRPO) with PPO for language-model post-training. Explain GRPO's advantages and limitations, and assess when its reward signal is sparse rather than dense.
### Constraints
Use the standard group-relative formulation as the baseline. Distinguish outcome-supervised GRPO from process-supervised variants; do not assume that GRPO inherently permits only one reward type. No particular model or benchmark result is required.
### Clarifying Questions
- Is one reward given for the whole response, or are intermediate reasoning steps scored?
- How many responses are sampled for each prompt, and how often do their rewards differ?
```hint A repeated signal is not a new observation
Applying one response-level score to many tokens does not create independent feedback about which intermediate step was correct.
```
### What a Strong Answer Covers
- Group-relative advantage estimation and the removal of a separately trained critic.
- Sampling cost, reward variation, memory tradeoffs, and credit-assignment limitations.
- Sparse outcome rewards versus denser process supervision, with qualified claims.
### Follow-up Questions
- What happens when every response in a group receives the same reward?
- How could an unreliable reward function affect both GRPO and PPO?
Overview: Explain GRPO group-relative advantages, critic-free tradeoffs, reward variation, and the difference between outcome and process supervision.
Read the full Siemens Machine Learning Engineer interview experience this question came from
Community answers
Answer by leon.ericssonabb
PPO uses a learned critic V(st)V(s_t) to estimate token-level advantages, typically with GAE. GRPO removes the critic: for each prompt, it samples GG responses and computes advantages relative to the group, e.g.
Ai=Ri−μRσR+ϵ.A_i=\frac{R_i-\mu_R}{\sigma_R+\epsilon}.These advantages are then used in a PPO-style clipped objective.
GRPO's main benefit is eliminating the critic, reducing memory, compute, and implementation complexity. Its main cost is sampling multiple responses per prompt, and its learning signal depends on reward variation within the group. If every response gets the same reward, the relative advantage is essentially zero.
With outcome-supervised GRPO, each response gets one terminal reward, and that response-level advantage is typically applied to all its tokens. This is still a sparse reward signal: repeating the same advantage across tokens does not create new credit-assignment information. GRPO knows which response was better, but not which reasoning steps caused the difference.
GRPO is not inherently limited to outcome rewards. Process-supervised GRPO can score intermediate steps, producing denser supervision and better temporal credit assignment.
So GRPO trades PPO's learned critic for more rollout sampling and stronger dependence on within-group reward variation. PPO can provide more state-specific credit assignment through its critic, but introduces critic training cost and value-estimation error. Both methods remain dependent on reward quality; an unrel