LLM Evaluation And Product Understanding

What's being tested

Interviewers are testing whether you can evaluate LLM product quality as a Data Scientist, not whether you can design the model architecture or serving stack. The core skill is translating fuzzy product goals like “helpful,” “safe,” or “engaging” into measurable outcomes, then choosing the right mix of offline evaluation, human judgment, and online experimentation. Meta cares because LLM features in `Messenger`, `Instagram`, `Facebook`, and `WhatsApp` can create user value but also introduce risks: hallucination, low trust, safety regressions, and misleading engagement. A strong answer shows you can balance user impact, statistical validity, guardrail metrics, and product context.

Core knowledge

Product success metrics should start from the user job-to-be-done. For an AI assistant, possible primary metrics include `task_completion_rate`, `helpfulness_rating`, `repeat_usage_7d`, `conversation_success_rate`, or reduction in `time_to_resolution`, depending on whether the product is search, support, creation, or chat.
Offline evaluation is useful before launch but rarely sufficient. Common offline metrics include exact match, semantic similarity, factuality scores, toxicity scores, retrieval precision, and rubric-based human ratings. For open-ended generations, `BLEU` and `ROUGE` are often weak proxies because many valid answers differ lexically.
Human evaluation is central for subjective dimensions like helpfulness, coherence, tone, and safety. Use blinded side-by-side comparisons, randomized answer order, calibrated rubrics, and multiple raters per item. Track inter-rater reliability with Cohen’s $\kappa$ or Krippendorff’s $\alpha$ when labels are categorical.
Pairwise preference testing often beats absolute scoring because raters are better at choosing “A vs. B” than assigning a 1–5 score. Aggregate comparisons using Bradley-Terry or Elo-style models: $P(A \succ B)=\frac{e^{s_A}}{e^{s_A}+e^{s_B}}$ where $s_A$ and $s_B$ are latent quality scores.
LLM-as-judge can reduce evaluation cost but must be calibrated against human labels. Measure judge-human agreement, bias toward longer answers, position bias, verbosity bias, and sensitivity to prompt wording. Treat it as a noisy measurement instrument, not ground truth.
Hallucination measurement requires separating factuality from usefulness. A response can be fluent and helpful-looking but unsupported. Metrics can include `unsupported_claim_rate`, `citation_precision`, `answer_abstention_accuracy`, and human-rated factual correctness. For retrieval products, evaluate whether claims are supported by retrieved evidence.
Safety and integrity guardrails should be evaluated separately from engagement. Track `policy_violation_rate`, toxic output rate, self-harm handling quality, misinformation risk, privacy leakage, and harmful compliance. A launch should not optimize `sessions_per_user` while increasing unsafe outputs.
Online experimentation should use randomized controlled trials when feasible. Define a primary metric, guardrails, MDE, duration, and stopping rule before launch. For binary outcomes, approximate sample size per arm is $n \approx \frac{2(z_{\alpha/2}+z_\beta)^2p(1-p)}{\delta^2}$ where $\delta$ is the desired detectable lift.
Novelty effects are common in AI products. Early `DAU` or `messages_per_user` lift may reflect curiosity rather than durable value. Include retention metrics such as `D7_retention`, repeat successful sessions, or cohort-level usage after first exposure.
Segment analysis matters because LLM quality varies by language, region, query type, age of content, and user intent. For Meta-scale products, always inspect slices like locale, device, new vs. existing users, high-risk topics, and creators vs. consumers before declaring a win.
Counterfactual logging and selection bias affect interpretation. If only users who opt into an AI assistant are measured, results may not generalize. In online tests, randomize exposure eligibility when possible; in observational analysis, be explicit about confounding and avoid causal claims from self-selected usage.
Metric gaming is especially dangerous with chat systems. Longer sessions may mean engagement, confusion, or failure. More messages can indicate delight or inability to complete the task. Pair engagement metrics with quality metrics like `task_success_rate`, `user_satisfaction`, and negative feedback rate.

Worked example

Example prompt: “How would you evaluate an LLM-powered assistant in Messenger?”

A strong candidate would start by clarifying the assistant’s purpose: is it helping users draft messages, answer questions, plan activities, or provide customer support? They would also ask whether the feature is opt-in, which markets are included, and what risks are most important, such as hallucination, privacy, or unsafe advice. The answer should then be organized around four pillars: product goal and primary metric, offline quality evaluation, online experiment design, and guardrail/segment analysis. For example, if the assistant helps users complete tasks in chat, the primary metric might be `task_completion_rate` or post-session `helpfulness_rating`, not raw `messages_sent`. Offline, they would propose human-rated prompt sets covering common intents, long-tail intents, and safety-sensitive prompts, with blinded pairwise comparisons against the current baseline. Online, they would run an A/B test with user-level randomization, measuring `D7_repeat_usage`, `thumbs_down_rate`, `conversation_abandonment_rate`, and safety violation rates. A key tradeoff to flag is that optimizing for engagement could reward addictive or frustrating interactions, so quality-adjusted engagement is better than volume alone. They should close by saying that, with more time, they would build a slice-based dashboard to monitor quality by language, topic, and user cohort, because aggregate metrics can hide serious regressions.

A second angle

Example prompt: “How would you measure whether an LLM is hallucinating?”

The same evaluation principles apply, but the primary challenge shifts from product engagement to factual correctness and evidence support. A strong answer would define hallucination operationally: unsupported claims, contradicted claims, fabricated citations, or overconfident answers when the model should abstain. The evaluation set should be stratified by topic difficulty, freshness, language, and risk level, because hallucination rates are not uniform. Instead of relying on user engagement, the candidate should propose human fact-checking, retrieval-grounded support labels, `unsupported_claim_rate`, and calibration metrics for confidence or refusal behavior. The key tradeoff is precision versus coverage: a highly conservative model may hallucinate less but refuse too often, hurting usefulness.

Common pitfalls

Pitfall: Treating engagement as the only success metric.

A tempting answer is “measure increase in `DAU`, session length, and messages sent.” That is incomplete because more usage can mean confusion, novelty, or failure. A better answer pairs engagement with outcome quality: `task_success_rate`, satisfaction, negative feedback, retention, and safety guardrails.

Pitfall: Saying “use human evaluation” without specifying the design.

Human eval is not automatically reliable. Strong candidates describe blinded comparisons, clear rubrics, multiple raters, representative prompt sampling, inter-rater reliability, and how labels map to launch decisions. Otherwise, the interviewer may see the answer as hand-wavy.

Pitfall: Over-indexing on generic NLP metrics.

Metrics like `BLEU`, `ROUGE`, or embedding similarity can be useful in narrow summarization or translation tasks, but they often fail for open-ended assistants. For product evaluation, it is better to use task-specific rubrics, pairwise preference, factuality checks, and online user outcomes.

Connections

This topic often pivots into A/B testing, metric design, causal inference with self-selection, and ranking/recommender evaluation. If the interviewer pushes deeper, expect discussion of heterogeneous treatment effects, guardrail metrics, sequential testing, or how to reconcile offline model wins with online product regressions.