Evaluate an ads algorithm change
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
Your ads team has developed a new ad-ranking algorithm for feed delivery. The new algorithm is expected to improve ad relevance and monetization, but it may also hurt user experience, shift delivery toward a small set of advertisers, or create short-term gains that do not hold long term.
How would you decide whether to launch this algorithm?
In your answer, discuss:
- the primary business objective and why a single metric such as CTR is insufficient
- the experiment design, including randomization unit and rollout strategy
- key success metrics and guardrail metrics for users, advertisers, and platform health
- how to handle interference, auction effects, novelty effects, and delayed conversions
- what segment analyses you would run across geography, device, advertiser size, or user cohorts
- how you would proceed if offline model metrics and online experiment results disagree
Quick Answer: This question evaluates competency in experiment design, causal inference, metric selection, product analytics, and evaluation of ad-ranking systems, including handling interference, auction effects, novelty and delayed conversions, segmentation, and reconciling offline and online model signals.