Evaluate Recommendation Feature with Historical Data Analysis
Overview
This question evaluates a data scientist's competence in offline recommendation evaluation, covering skills in causal inference, counterfactual estimation, metric selection, assumption articulation, and validation using historical logs.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
19
0
Offline Evaluation of a New Recommendation Feature
Scenario
You need to estimate the business value of a new recommendation-system feature using only historical data, before any live deployment.
Task
Describe how you would evaluate whether releasing this feature is a good or bad idea without running an A/B test. Specify:
- The analyses you would run and in what order
- The metrics you would use
- The assumptions required for each method
- How you would validate the conclusions
Constraints
- Use only historical logs.
- Do not propose live randomized experiments.
Hints
- Offline log replay / counterfactual (off-policy) evaluation
- Inverse propensity scoring (IPS), self-normalized IPS (SNIPS), doubly robust (DR)
- Uplift or propensity modeling
- Simulation with a user-response model
- Historical hold-out metrics (e.g., NDCG, recall) and their calibration to online outcomes
Solution
Login required
Show