Design an Experiment to Evaluate New ML Model
Overview
This question evaluates a data scientist's competencies in online experiment design, causal inference, metric selection, assessment of statistical and practical significance, identification and handling of heterogeneous treatment effects, data-quality and marketplace interference mitigation, and ethical rollout decision-making.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Experiment Design: Validating a New Ads Ranking Model
Context
You operate an ads platform with an existing recommender/ranking model. Engineers built a new ML ranker that is hypothesized to improve outcomes. You need to run an online controlled experiment (A/B) to validate performance, choose appropriate metrics, assess significance, and make a rollout decision, including handling heterogeneous effects and ethics.
Tasks
-
Experiment design
- What is your randomization unit, ramp plan, duration, and sample size approach?
- How will you guard against marketplace interference and data-quality issues?
-
Metrics
- Define 2–3 primary KPIs and a set of guard-rail metrics. Explain why each matters for users, advertisers, and the platform.
-
Practical significance
- An A/B test shows a +5% lift in CTR. How do you judge practical (business) significance vs. statistical significance?
-
Heterogeneous effects
- CTR doubles for Indian males aged 18–55. What might this indicate, and what next steps do you take to validate and respond?
-
Rollout decision
- If the test shows +5% CTR and +5% revenue, do you roll out globally? Explain your decision process, including trade-offs and ethical checks.
Solution
Login required
Show