Decide when CTR falls but revenue rises
Overview
This question evaluates understanding of A/B experimentation, auction-based ad metrics, metric decomposition for revenue impact, causal diagnostics for hidden heterogeneity (e.g., Simpson’s paradox), and executive-level results presentation within the Analytics & Experimentation domain.
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Ads-Ranking A/B Test: Decision, Decomposition, Diagnostics, and Exec Readout
Context
You ran a user-level A/B test of a new ads-ranking model. The treatment vs. control results are:
- CTR: −3.0% (p = 0.02)
- CPM: +6.0% (p = 0.04)
- Impressions per user: +1.5% (p = 0.08)
- RPM (revenue per thousand impressions): +4.0% (p = 0.05)
- Purchase conversion on click (CVR_on_click): −0.8% (p = 0.20)
Assume RPM is net revenue per 1,000 ad impressions. CPM is advertiser cost per 1,000 impressions. The auction mix likely includes CPC/CPA inventory, so eCPM is a function of bids and predicted outcomes.
Tasks
- Create a decision framework to recommend ship or hold using a north star metric (e.g., revenue or advertiser value) with guardrails (user experience, advertiser outcomes, integrity), and apply it to the given results.
- Quantify the net revenue delta per 1,000,000 impressions and decompose drivers in a waterfall using CPM, CTR, and CVR.
- Specify additional diagnostics (pacing, bid landscape shifts, supply mix, user segments) that could reveal Simpson’s paradox or hidden heterogeneity.
- If presenting to the CFO, specify which visuals to include on one slide (e.g., forest plot of segment effects with CIs, waterfall of drivers, traffic allocation/SRM chart) and provide the headline.
Solution
Login required
Show