Decide when CTR falls but revenue rises

Q: Decide when CTR falls but revenue rises

This question evaluates understanding of A/B experimentation, auction-based ad metrics, metric decomposition for revenue impact, causal diagnostics for hidden heterogeneity (e.g., Simpson’s paradox), and executive-level results presentation within the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Ads-Ranking A/B Test: Decision, Decomposition, Diagnostics, and Exec Readout

Context

You ran a user-level A/B test of a new ads-ranking model. The treatment vs. control results are:

CTR: −3.0% (p = 0.02)
CPM: +6.0% (p = 0.04)
Impressions per user: +1.5% (p = 0.08)
RPM (revenue per thousand impressions): +4.0% (p = 0.05)
Purchase conversion on click (CVR_on_click): −0.8% (p = 0.20)

Assume RPM is net revenue per 1,000 ad impressions. CPM is advertiser cost per 1,000 impressions. The auction mix likely includes CPC/CPA inventory, so eCPM is a function of bids and predicted outcomes.

Tasks

Create a decision framework to recommend ship or hold using a north star metric (e.g., revenue or advertiser value) with guardrails (user experience, advertiser outcomes, integrity), and apply it to the given results.
Quantify the net revenue delta per 1,000,000 impressions and decompose drivers in a waterfall using CPM, CTR, and CVR.
Specify additional diagnostics (pacing, bid landscape shifts, supply mix, user segments) that could reveal Simpson’s paradox or hidden heterogeneity.
If presenting to the CFO, specify which visuals to include on one slide (e.g., forest plot of segment effects with CIs, waterfall of drivers, traffic allocation/SRM chart) and provide the headline.

Decide when CTR falls but revenue rises

Quick Overview

Ads-Ranking A/B Test: Decision, Decomposition, Diagnostics, and Exec Readout

Context

Tasks

Solution

Comments (0)