Evaluate New Ad Model with A/B Testing Experiment
Overview
This question evaluates experimental-design and product-analytics competencies—A/B testing, traffic allocation and ramping, metric definition and guardrails, statistical power and significance, and handling marketplace/auction interference—within the Analytics & Experimentation domain for a Data Scientist role.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Evaluate a New Ads Recommendation Model via Online Experimentation
Scenario
You have trained a new ad-recommendation model and must decide whether it should replace the incumbent model that currently ranks/serves ads in a large-scale, auction-based ads system.
Task
Design an experiment to evaluate the new model against the incumbent and answer:
- Experiment design: unit of randomization, traffic split/ramp, duration, and how to handle auction/marketplace interference.
- Metrics: define primary success metric(s) and guardrails (revenue, CTR, ROI, user retention, latency, etc.).
- Stakeholder readouts: how the story and metrics differ for a CFO vs a CGO (growth).
- Decision framework: launch or rollback criteria, with monitoring and risk mitigation.
Requirements
- Cover A/B design, traffic allocation, expected duration, statistical significance/power, and trade-offs between finance and growth.
- Include clear metric definitions and assumptions where needed.
Solution
Login required
Show