LLM Evaluation Interview Questions: Evals, LLM-as-Judge, Drift, and Production Quality

Your candidate model wins the offline eval by six points. After launch, users retry more often, support escalations rise, and the automated judge still says the new version is better. Which signal do you trust, and what do you investigate first?

That is the level of reasoning behind strong LLM evaluation interview answers. Interviewers are not looking for a memorized list of metrics. They want to see whether you can turn an ambiguous quality goal into representative tests, trustworthy graders, production monitoring, and a defensible ship-or-rollback decision.

Start with PracHub's real agent evaluation system question, then practice LLM judge versus human judge calibration. Use the broader interview question bank with written solutions to connect evaluation decisions to ML system design, experimentation, and production debugging.

Quick Answer: An Eval Is a Decision System, Not a Score

A production eval should answer a concrete question such as, "Can version B replace version A for support traffic without increasing critical errors, latency, or cost per resolved case?" The score matters only after the decision, population, rubric, and acceptable risk are defined.

How to Structure an LLM Evaluation Interview Answer

Use this order: decision, behavior, data, graders, uncertainty, action. First name the release or product decision. Then decompose "quality" into observable behaviors, build a dataset that covers the relevant population, select the cheapest valid grader for each behavior, quantify uncertainty, and define what each result triggers.

This order prevents a common mistake: choosing an attractive metric before deciding what it is supposed to represent. OpenAI's evaluation best-practices guide similarly recommends task-specific evals, production-like data, continuous evaluation, and human calibration of automated scoring.

Eval Design and Dataset Interview Questions

1. Design an eval suite for a customer-support assistant

Begin with business outcomes: correct resolution, safe escalation, policy compliance, and low user effort. Convert those outcomes into a failure taxonomy such as wrong answer, unsupported claim, missed escalation, invalid tool action, disclosure of restricted data, and excessive back-and-forth.

Use deterministic checks for schema, required fields, citation existence, and tool arguments. Use rubric-based review for correctness and helpfulness, then preserve human review for high-risk cases and disputed judgments. Track latency and cost alongside quality because a version that is marginally better but too slow or expensive may still fail the release decision.

2. How do you build a representative evaluation set?

Sample from real logs after applying privacy controls, then stratify by task, locale, customer tier, conversation length, risk, and tool use. Add known failures, rare edge cases, and adversarial prompts instead of letting frequent easy traffic dominate the average.

Keep a frozen release set for comparable regression testing, but also maintain rotating and hidden sets. New production failures should enter a reviewed regression pool. This makes the suite both stable enough to compare versions and capable of learning from the product.

3. When should you use exact match, semantic metrics, or human judgment?

Use exact or programmatic checks when correctness has a closed form: valid JSON, a known calculation, a tool call, a required citation, or a database result. These graders are cheap, repeatable, and easy to debug.

Similarity metrics can help when reference overlap is meaningful, but valid open-ended answers may use different wording. For nuanced correctness, usefulness, tone, or policy interpretation, use a tightly defined model rubric, human judgment, or both. The key is grader-task fit, not one universal metric.

4. How do you prevent benchmark overfitting and contamination?

Separate development, validation, and final release sets; restrict access to hidden cases; version every dataset; and record whether tools can retrieve benchmark answers. Do not optimize repeatedly against one public score and then present it as generalization.

Track performance on fresh production samples and newly discovered failures. If the visible benchmark improves while hidden or rotating sets do not, treat the gain as suspicious until you understand why.

LLM-as-a-Judge Interview Questions

5. When is LLM-as-a-Judge appropriate?

A model judge is useful when outputs are open-ended, the rubric can be expressed clearly, and human review of every sample would be too slow. It can label broad test sets quickly and explain which criterion failed.

It is not ground truth. Avoid asking a judge to infer facts it cannot verify, grade reasoning beyond its capability, or collapse several ambiguous dimensions into one unexplained score. Prefer deterministic evidence whenever the behavior can be checked directly, and retain human adjudication for critical or uncertain cases.

6. Pairwise comparison or absolute scoring?

Pairwise comparison asks which of two outputs better satisfies a rubric. It often fits regression decisions because the real question is whether candidate B is better than baseline A. Absolute scoring is easier to store and trend, but a 1-to-5 scale can shift when the judge model, prompt, or examples change.

For pairwise judging, randomize order, evaluate both orders on sensitive cases, allow ties, and report the tie policy. For absolute scoring, anchor every score with concrete examples and test whether small prompt changes alter the distribution.

7. What biases affect LLM judges?

Important failure modes include position bias, verbosity preference, style preference, and possible self-preference. A fluent answer can receive a high score even when it misses a requirement, and a judge can reverse a pairwise decision after the response order is swapped.

The original MT-Bench and Chatbot Arena judge study documents position, verbosity, and self-enhancement concerns. The separate FairEval study demonstrates order sensitivity and evaluates calibration strategies. Mitigations include order swapping, explicit criterion-by-criterion grading, reference evidence, adversarial judge tests, and human review of disagreements.

8. How do you calibrate an LLM judge against humans?

Create a human-labeled validation set that covers the important slices and includes borderline examples. Lock the rubric, collect independent ratings where risk warrants it, adjudicate disagreements, and compare the automated judge with the resulting labels.

Report per-class precision and recall for categorical decisions, or agreement and error by score band for ordinal ratings. Inspect disagreement examples rather than celebrating one aggregate correlation. Recalibrate whenever the judge model, judge prompt, rubric, task distribution, or output format changes.

Drift and Production Quality Interview Questions

9. What does LLM quality drift mean?

Quality drift is a change in user-relevant behavior over time. It may come from a model update, but it can also come from changing traffic, a new prompt, a stale retrieval index, tool failures, policy changes, or a drifting judge.

That distinction matters because "the model got worse" is not yet a diagnosis. Version the model, prompts, tools, retrieval corpus, policies, datasets, and graders so the team can reconstruct what changed.

10. Offline quality improved, but production quality fell. How do you debug it?

First verify the measurement pipeline: logging, sample selection, labels, dashboards, and judge versions. Next compare production traffic with the eval distribution, then replay the same examples through old and new system versions while holding dependencies constant.

Slice the regression by task, locale, length, risk, retrieval state, and tool path. Inspect complete traces, not only final text. A strong root-cause tree separates model behavior from input-mix drift, retrieval freshness, tool errors, prompt changes, and evaluator drift.

11. What do you monitor before ground-truth labels arrive?

Use leading signals tied to a known failure hypothesis: schema failures, tool error rate, citation validity, retrieval coverage, refusal rate, repeated questions, user corrections, escalation, abandonment, latency, and cost per successful task. None is a universal quality metric; each is a proxy with a documented limitation.

Sample live traffic for delayed model and human review, and join those labels back to the original trace. NIST's Generative AI Profile recommends post-deployment monitoring that incorporates user input, incident response, recovery, and change management.

12. How do you detect evaluator drift?

Maintain a fixed judge-control set with human labels, known ties, adversarial verbosity cases, and order-swapped pairs. Run it whenever the judge model, prompt, rubric, or output schema changes, and periodically even when no change is planned.

Track agreement by slice, score distribution, abstention or invalid-output rate, and order consistency. If the judge moves while human labels do not, pause trend comparisons or rescore historical outputs with the new judge under a documented migration.

13. How would you define a release gate?

Use one primary outcome plus guardrails. Compare candidate and baseline on the same examples, estimate uncertainty, require no regression on critical slices, and inspect new failure modes before shipping. For stochastic agents, repeat trials and evaluate trajectories as well as final answers.

A good gate ends in an action: ship, hold for review, canary to a limited cohort, or roll back. Thresholds belong to the product's risk tolerance and baseline; they should never be copied blindly from another system.

A Complete LLM Evaluation Walkthrough

Suppose a new support agent scores six points higher offline, but production correction and escalation rates rise. Freeze the rollout and confirm that those metrics are computed correctly. Pull a stratified sample of affected traces, replay both versions against the same tool and retrieval snapshots, and have calibrated reviewers label the failure taxonomy.

The analysis reveals that the new model writes more polished answers, so the LLM judge prefers it, but it also attempts a refund tool before confirming eligibility. The static eval contains few refund cases, and its rubric rewards helpfulness without a separate tool-policy criterion.

The fix is not simply "use a better judge." Add deterministic tool-policy checks, expand the refund slice, split helpfulness from procedural correctness, recalibrate the judge, and make production escalation a delayed outcome metric. An illustrative release policy might look like this:

release_gate:
  primary:
    resolved_without_correction_delta: ">= +2 percentage points"
  guardrails:
    critical_tool_error_delta: "<= +0.1 percentage points"
    p95_latency_delta: "<= +10%"
    cost_per_success_delta: "<= +5%"
  evidence:
    paired_confidence_interval: "lower bound > 0"
    human_audit: "no new critical failure mode"
  action_on_failure: "hold or roll back"

These thresholds are examples, not industry standards. The interview signal is that the candidate connects evidence to a decision, identifies who owns the decision, and can explain what happens when metrics disagree.

How Interviewers Score LLM Evaluation Answers

A Focused 5-Day Preparation Plan

For a concrete system, practice designing a RAG system with component and end-to-end evaluation. Then use PracHub's LLM evaluation and observability concept guide to review the vocabulary behind your decisions.

Frequently Asked Questions

Are BLEU or ROUGE enough for LLM evaluation?

No. They can be useful when overlap with a reference is meaningful, but open-ended outputs may be correct without matching the reference wording. Combine task-specific programmatic checks, rubric-based evaluation, and human calibration based on the behavior and risk being measured.

Can LLM-as-a-Judge replace human evaluation?

Not categorically. A calibrated model judge can scale routine scoring, while humans define the rubric, label validation data, adjudicate ambiguity, audit high-risk cases, and discover new failure modes. The right design is often hybrid and changes with task risk and judge reliability.

How many examples should an LLM eval contain?

There is no universal number. Required sample size depends on baseline rate, variance, minimum meaningful change, slice coverage, and the cost of a wrong release decision. Show the interviewer how you would estimate uncertainty and add data where critical slices remain underpowered.

What is the biggest LLM evaluation interview mistake?

Starting with a metric instead of a decision. First define who is affected, what failure matters, and what action the evidence will trigger. Only then choose the dataset, grader, statistical test, production signal, and threshold.

Practice Making the Release Decision

The strongest candidate can explain not only how to calculate a score, but why the score represents the intended behavior, when it can lie, and what the team should do next. That is the difference between running a benchmark and operating a reliable AI product.

Use PracHub to practice real interview questions with written solutions, then add company-specific interview prep for your target loop. Senior candidates should also rehearse the incident story through behavioral and leadership interview practice: how you found the bad signal, challenged a misleading metric, and changed the release process.

Sources

• OpenAI: Evaluation Best Practices
• OpenAI: Graders
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Wang et al.: Large Language Models Are Not Fair Evaluators
• Stanford CRFM: Holistic Evaluation of Language Models
• NIST AI 600-1: Generative Artificial Intelligence Profile