Debug a GRPO training loop and explain ratios

You are given a simplified implementation of a GRPO (Group Relative Policy Optimization) training step for an RLHF-style policy model. The training is supposed to be strictly on-policy, meaning rollouts are generated by the same policy that is being updated.

Tasks:

1. Walk through the end-to-end GRPO training flow : sampling prompts, generating rollouts, computing group-based advantages (relative within a group of completions), computing the policy gradient loss, and updating the policy.
2. You find that training is unstable due to several straightforward implementation issues. Describe three common, easy-to-miss bugs in a GRPO/PPO-like training loop that would cause incorrect learning (e.g., wrong log-prob computation, incorrect masking, advantage normalization mistakes, mixing policies, etc.). For each, explain how you would detect it and how to fix it.
3. During debugging you notice the importance-sampling ratio

$\text{ratio} = \exp(\log \pi_{\theta}(a|s) - \log \pi_{\text{old}}(a|s))$

is not always 1, even though you expected the method to be strictly on-policy. Explain why the ratio might deviate from 1 in practice. List the most likely causes and what to check in the training pipeline to confirm each cause.