Estimate ads ranking revenue impact
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You are the data scientist for an ads ranking team. The team has built a new ranking algorithm for feed ads. The new model changes the ordering of ads by combining bid, predicted click-through rate, predicted conversion rate, and ad quality differently from the current production ranker.
A short ramp suggests that revenue per daily active user increased, but the team is worried that the short-term lift may not represent the medium-term impact because users may adapt, advertisers may change bids or budgets, and auction dynamics may shift.
Design an approach to estimate the medium-term revenue impact of launching the new ads ranking algorithm over a 4- to 8-week horizon. Address:
1. What is the causal estimand?
2. What experiment or quasi-experiment would you run?
3. What primary, secondary, and guardrail metrics would you track?
4. How would you handle auction interference, advertiser budget constraints, seasonality, and user-level heterogeneity?
5. How would you translate the experiment result into an estimated company-level revenue impact?
6. What launch recommendation would you make if short-term revenue is positive but some user-experience guardrails worsen?
Quick Answer: This question evaluates causal inference and experimentation skills, metric design, and revenue-impact analysis within ad auction systems, focusing on handling auction interference, advertiser behavior, seasonality, and user heterogeneity in an Analytics & Experimentation (Data Scientist) context.