Evaluate New Ad Model with A/B Testing Experiment
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
As part of the Ads team you have trained a new ad-recommendation model and need to decide if it should replace the current model.
##### Question
Design an experiment to evaluate the new recommendation model against the incumbent.
What primary and guardrail metrics would you track? How would these differ when presenting results to the CFO versus the CGO (growth)?
State your final recommendation framework for launch or rollback.
##### Hints
Cover experiment design (A/B, traffic split, duration), metrics (revenue, CTR, ROI, user retention), statistical significance and trade-offs for finance vs growth.
Quick Answer: This question evaluates experimental-design and product-analytics competencies—A/B testing, traffic allocation and ramping, metric definition and guardrails, statistical power and significance, and handling marketplace/auction interference—within the Analytics & Experimentation domain for a Data Scientist role.