Design an Experiment to Evaluate New Recommendation Model
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Meta’s ads platform team claims a newly-built ML ranking model will outperform the current recommendation system.
##### Question
How would you design an experiment to evaluate the new recommendation model? Specify control / treatment assignment, duration and success metrics. Which primary, guard-rail and long-term metrics would you choose and why? If an A/B test shows a 5 % lift in overall CTR, how would you quantify the business gain? The test reports a 100 % CTR increase for males aged 18-55 in India. How do you interpret this result? If the experiment shows +5 % CTR and +5 % revenue with no negative guard-rail impact, would you roll out the model? Explain.
##### Hints
Discuss sample size, statistical significance, heterogeneity, north-star vs. guard-rail metrics and risk mitigation.
Quick Answer: This question evaluates proficiency in online experiment design, A/B testing, metric selection and interpretation, statistical power and significance, heterogeneity analysis, and translating CTR lifts into quantified business impact.