Decide when CTR falls but revenue rises
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
You run an ads-ranking A/B. Results (treatment vs control, user-level): CTR −3.0% (p=0.02), CPM +6.0% (p=0.04), impressions/user +1.5% (p=0.08), revenue per thousand impressions (RPM) +4.0% (p=0.05), purchase conversion on click −0.8% (p=0.20). (1) Create a decision framework to recommend 'ship/hold' using a north star (e.g., revenue or advertiser value) and guardrails (user experience, integrity). (2) Quantify the net revenue delta per 1M impressions and decompose drivers in a waterfall (CPM, CTR, CVR). (3) Specify additional diagnostics (pacing, bid landscape shifts, supply mix, user segments) to catch Simpson’s paradox. (4) If presenting to the CFO, which visuals do you include on one slide (e.g., forest plot of segment effects with CIs, waterfall of drivers, traffic allocation/SRM chart), and what headline would you use?
Quick Answer: This question evaluates understanding of A/B experimentation, auction-based ad metrics, metric decomposition for revenue impact, causal diagnostics for hidden heterogeneity (e.g., Simpson’s paradox), and executive-level results presentation within the Analytics & Experimentation domain.