Design an Experiment to Evaluate New Recommendation Model
Overview
This question evaluates proficiency in online experiment design, A/B testing, metric selection and interpretation, statistical power and significance, heterogeneity analysis, and translating CTR lifts into quantified business impact.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
111
0
Loading...
Experiment Design: New Ads Ranking Model vs. Current System
Context
You are evaluating a newly built ML ranking model for an ads recommendation surface. The goal is to determine whether the new model should replace the current system based on rigorous online experimentation.
Task
Design an experiment to evaluate the new recommendation model. Specify:
- Control and treatment assignment, traffic split, contamination avoidance, and duration.
- Primary, guard-rail, and long-term metrics (with rationale) and success criteria.
- Sample size and statistical significance considerations; address heterogeneous effects.
- How to quantify business gain if an A/B test shows a 5% lift in overall CTR.
- Interpretation of a reported 100% CTR increase for males aged 18–55 in India.
- Decision-making: If the experiment shows +5% CTR and +5% revenue with no negative guard-rail impact, would you roll out? Explain risk mitigation.
Solution
Login required
Show