Evaluate Recommendation Feature with Historical Data Analysis

Q: Evaluate Recommendation Feature with Historical Data Analysis

Meta recommendation analytics prompt on offline evaluation using historical data, covering replay, IPS, SNIPS, doubly robust estimators, offline ranking metrics, assumptions, overlap, validation, and launch caveats.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Offline Evaluation of a Recommendation Feature With Historical Data

The company is considering launching a new recommendation-system feature and wants to judge its value before any live deployment.

Use only historical logs. Do not answer "run an A/B test."

Constraints & Assumptions

Historical logs were generated by an existing recommendation policy.
Offline evaluation may be biased if the new feature recommends items not sufficiently explored in historical data.
State the assumptions behind each method.
Validate conclusions with multiple approaches rather than one offline metric.

Clarifying Questions to Ask Guidance

What outcome matters: CTR, watch time, conversion, revenue, retention, diversity, or safety?
Were propensities or ranking scores logged?
Did the historical system include random exploration?
How different is the proposed recommendation policy from the historical policy?

What a Strong Answer Covers Guidance

Define unit of analysis, target outcomes, guardrails, and decision criteria.
Audit historical data for logging policy, propensities, feature availability at decision time, overlap/support, missing data, and leakage.
Offline metrics such as recall@K, NDCG, MAP, calibration, diversity, coverage, novelty, and segment performance.
Counterfactual/off-policy methods such as replay, inverse propensity scoring, self-normalized IPS, doubly robust estimation, uplift/propensity modeling, and simulations.
Assumptions: positivity/overlap, correct propensity logging, no unobserved confounding for observational approaches, stable user response, and SUTVA limitations.
Validation: backtesting against past launches, sensitivity analysis, bootstrap CIs, segment checks, placebo policies, and comparison to business outcomes.
Clear caveat that offline evidence can support a launch decision but cannot fully replace live causal measurement.

Follow-up Questions Guidance

What if the new policy recommends items never shown historically?
Why can offline NDCG fail to predict online lift?
How would you use propensities if they were logged?
What threshold would justify a limited rollout?

Evaluate Recommendation Feature with Historical Data Analysis

Quick Overview

Evaluate Recommendation Feature with Historical Data Analysis

Offline Evaluation of a Recommendation Feature With Historical Data

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer