LLM Post-Training Interview Questions: SFT, DPO, PPO, and GRPO Trade-Offs
Quick Overview
Prepare for LLM post-training interviews by comparing SFT, DPO, PPO, and GRPO through supervision, objectives, sampling, and evaluation. Work through checked preference-loss and advantage examples, distinguish paper formulations from TRL defaults, and diagnose reward failures.
LLM post-training interview questions become easier to reason through when you start with the supervision available. Demonstrations, preference pairs, and rewards on newly sampled outputs support different training choices. SFT, DPO, PPO, and GRPO are not a ladder from weakest to strongest. Connect each objective to its data, compute requirements, and a failure you can detect.
This guide separates primary research and official documentation from original numerical exercises and preparation recommendations. It does not claim a standard employer interview loop. The linked PracHub LLM interview questions are reported practice material across roles, not predictions of your next prompt.
How Would You Choose a Post-Training Method?
Preparation framework: Ask what the model cannot do, what trustworthy supervision exists, and whether you can afford repeated generation and evaluation. A team with excellent demonstrations and weak preference labels faces a different problem from one with a reliable executable verifier.
| Method | Training signal in the setup discussed here | Question to resolve first |
|---|---|---|
| SFT | Demonstrated target responses | Are the demonstrations correct and representative? |
| Original DPO | Preferred/rejected responses for the same prompt | Do the pairs encode the behavior you want? |
| PPO-based RLHF | Sampled outputs scored by a reward model, with value estimates | Can the reward and training loop be trusted? |
| Outcome-supervised GRPO | Rewards compared within groups of sampled responses | Do groups contain informative reward differences? |
These are compatible stages or alternatives in a larger recipe. A choice should be conditional: “I would establish an SFT baseline, then evaluate preference optimization if the remaining problem is response selection.” Avoid recommending an expensive online loop before identifying what new information its rollouts would provide.
What Does SFT Optimize, and What Can It Miss?
Official implementation fact: Supervised fine-tuning trains on examples using a language-model loss. TRL supports different dataset formats and loss masks, including completion-only or assistant-only training in applicable configurations. Check which tokens receive loss rather than assuming every prompt and response token is treated identically. TRL SFT Trainer
For an interview, describe teacher forcing: the model predicts target tokens given the prompt and preceding target tokens. A typical response-only loss is the negative sum or average of target-token log probabilities, with padding and excluded positions masked.
Preparation inference: If demonstrations contain unsupported claims or formatting mistakes, fitting them more closely does not solve the underlying data problem. Inspect examples, task coverage, and evaluation leakage before attributing a disappointing result to the optimizer.
SFT also does not directly express “response A is better than response B” unless you transform that information into a training scheme. Selecting only winners discards some comparative information. Whether that matters should be evaluated against a clear baseline, rather than assumed from the method's name.
Why Does DPO Avoid a Separate Reward Model?
Primary research: Original Direct Preference Optimization reparameterizes a KL-regularized reward objective through the policy and reference policy. Under its preference-model assumptions, the prompt-specific normalization term cancels in a pairwise comparison. The result is a classification loss over preferred and rejected responses, without a separately fitted reward model or online RL rollout loop during the original offline training procedure. DPO paper, Section 4
For one prompt, write the loss using natural-log sequence probabilities:
m = (log p(chosen) - log pref(chosen))
- (log p(rejected) - log pref(rejected))
L = -log sigmoid(beta * m)
Here p is the trainable policy, pref is a fixed reference, and beta is positive. These are conditional completion probabilities for the same prompt. Explain the reference-relative margin; saying “DPO increases the chosen probability” misses an important qualification.
Official implementation fact: TRL's DPO documentation expects preferred and dispreferred completions, and exposes multiple loss variants. Its original sigmoid objective should not be conflated with every option available under DPOTrainer. TRL DPO Trainer
DPO still depends on preference quality. “No separate reward model” does not mean no supervision, no assumptions, or immunity to noisy labels.
Work Through a Preference Pair Numerically
Original exercise: Let reference log probabilities be −2 for the chosen response and −3 for the rejected response. Initially, the policy equals the reference. The margin is zero and the loss is log(2), approximately 0.693147.
Set beta = 0.5. Policy A assigns log probabilities −1.5 and −3.5. Its margin is (−1.5 + 2) − (−3.5 + 3) = 1, giving a loss of approximately 0.474077.
Policy B assigns −2.5 and −4.5. Its margin and loss are unchanged, but its chosen-response probability is lower than the reference: approximately 0.082 instead of 0.135. Policy A's chosen probability is approximately 0.223. The remaining probability mass belongs to other possible responses.
This is a counterexample about objective values, not a claim that a particular optimizer step must produce Policy B. A pair's lower loss does not prove that its preferred response became more likely in absolute terms.
The local calculation below reproduces both losses. Inputs are sequence log probabilities, not raw logits or tokenwise averages.
import math
def dpo_loss(pc, pr, rc, rr, beta=0.5):
margin = (pc - rc) - (pr - rr)
z = -beta * margin
return max(z, 0.0) + math.log1p(math.exp(-abs(z)))
print(round(dpo_loss(-2, -3, -2, -3), 6))
print(round(dpo_loss(-1.5, -3.5, -2, -3), 6))
print(round(dpo_loss(-2.5, -4.5, -2, -3), 6))
# 0.693147, 0.474077, 0.474077
Technical follow-up: The derivative with respect to m is −beta * sigmoid(−beta*m). Increasing the margin lowers this loss. That scalar statement is different from predicting every parameter change in a shared neural network.
What Does PPO Clip, and Which Policy Is “Old”?
Primary research: PPO's clipped surrogate uses a current-to-old policy probability ratio and an estimated advantage. For a sampled action, it maximizes the smaller of the unclipped and clipped terms. Clipping limits the incentive for certain large changes; it does not impose a hard bound on every policy probability or guarantee improvement. PPO paper, Equation 7
ratio = exp(log p_current(action) - log p_old(action))
objective = min(ratio * advantage,
clip(ratio, 1-epsilon, 1+epsilon) * advantage)
Original check: With advantage 2, ratio 1.5, and epsilon 0.2, the terms are 3.0 and 2.4, so the surrogate is 2.4. With advantage −2 and ratio 0.5, they are −1.0 and −1.6, so the surrogate is −1.6. The sign matters; blindly clipping a ratio and dropping the min changes the objective.
In a PPO-based language-model RLHF setup, distinguish three roles. The old policy generated the rollout and supplies the fixed denominator for that batch. The reference policy anchors a separate drift penalty. The value model, or critic, estimates returns to help form advantages. It is not the reward model that scores output quality. TRL PPO Trainer
A debugging answer should identify when ratios should be near one: before an update, with matching parameters, inputs, masks, and probability definitions. After updating against fixed old log probabilities, non-unit ratios are expected. Sampling transforms, stale weights, token shifts, and different scoring paths require separate checks.
What Does GRPO Remove, and What Does It Keep?
Primary research: DeepSeekMath introduced Group Relative Policy Optimization, which removes the additional value-model approximation and obtains a baseline from multiple responses to the same prompt. In its outcome-supervised formulation, each response's reward is centered and scaled using the group's rewards; that response-level advantage is applied across its tokens. The original objective also includes clipping and a reference-policy KL term. DeepSeekMath, Section 4.1
Original exercise: Four responses receive rewards [0, 0, 1, 1]. With population standard deviation, the mean is 0.5 and standard deviation is 0.5. Normalized advantages are approximately [−1, −1, 1, 1] when using a small denominator safeguard.
For [0, 0, 0, 0], centered rewards are zero. Dividing by std + epsilon yields zeros rather than NaNs. That group supplies no reward-based relative policy-gradient signal under this calculation. A separately active KL term may still contribute; “the entire training gradient is zero” would be too broad.
Removing the critic saves its associated work, but generating several responses per prompt still costs time and memory. GRPO also needs rewards: a learned model or a rule-based verifier can supply them. Critic-free is not reward-free.
How Would You Diagnose Reward Hacking?
Original failure case: A code-training verifier rewards responses that contain the expected final string. The model learns to print that string regardless of whether its code solves the task. Training reward rises while held-out execution accuracy stays flat.
First reproduce the mismatch on saved outputs. Compare the reward function's decision with an independent execution check; inspect false positives, malformed answers, and repeated templates. Keep evaluation tasks separate from training prompts and avoid rewarding access to the evaluator itself.
Next, repair the signal before choosing a new optimizer. Stronger tests, output parsing, and review of adversarial cases can make the reward more faithful. Switching PPO to GRPO will not fix a verifier that rewards the wrong behavior.
An offline preference dataset can fail similarly when its winners are consistently longer but not more correct. Inspect label rubrics and matched examples, then evaluate correctness and verbosity separately. A better preference margin cannot by itself establish better task performance.
How Do Offline and Online Training Change the Plan?
Preparation recommendation: With a fixed pair dataset and limited generation budget, start by comparing an SFT baseline with original offline DPO. State how the preference data was sampled, how ties were handled, and which deployment prompts it fails to cover.
With a trustworthy executable reward and a need to explore new solutions, consider an online rollout method. Explain why fresh samples could reveal useful behaviors absent from the fixed dataset. Budget for generation, reward evaluation, policy updates, and checking that rollout versions match the probabilities used in training.
For either plan, report held-out quality alongside training diagnostics. Include task success, response length, relevant safety failures, and evaluation uncertainty. Track reward components and policy drift to investigate failures; do not substitute them for user-facing quality.
A defensible recommendation names a baseline, the evidence favoring the next stage, and a stopping condition. For example: “I would stop scaling this run if verifier reward improves but independently checked correctness does not, and audit reward false positives before collecting more rollouts.”
Which TRL Details Should You Verify Before Coding?
Official documentation snapshot, checked September 8, 2026: TRL's release page lists v1.12.0 as its latest release. The documentation exposes PPO under trl.experimental.ppo, while DPO and GRPO appear among the main trainers. Treat that API distinction as an implementation detail, not a verdict on the underlying algorithms. TRL releases, PPO API
The current GRPO documentation exposes several loss formulations and lists dapo as the loss_type default. Consequently, “I used GRPOTrainer” does not identify the original paper's exact normalization. Record the installed version, loss type, reward scaling, KL coefficient, group size, and truncation policy. TRL GRPO Trainer
The exercises here validate scalar arithmetic, not a GPU training run or a tested TRL integration. In implementation questions, also inspect label shifting, completion masks, detached old/reference scores, and reward-group boundaries before tuning learning rates.
Practice Post-Training Reasoning With PracHub
These records span companies and software/ML engineering roles. Use them to practice technical reasoning; they do not establish a universal post-training interview syllabus.
| PracHub question | Practice focus |
|---|---|
| Compare Language-Model Post-Training Methods | Choose by supervision, objective, and evaluation constraints. |
| Explain DPO and construct its training data | Defend preference-pair quality and reference-relative reasoning. |
| Debug a GRPO training loop and explain ratios | Separate expected ratio changes from implementation bugs. |
| Explain Pipeline Parallelism in GRPO Training | Distinguish rollout generation from policy-update scheduling. |
| Design multimodal deployment under compute limits | Connect training choices to deployment and evaluation limits. |
Continue with LLM interview questions. Rework one numerical example, explain what changes, then propose an evaluation that could disprove your preferred training choice.
Comments (0)