Estimate Lift and Significance in Facebook Ad Campaigns
Overview
This question evaluates a data scientist's competence in estimating conversion lift and testing statistical significance for randomized online advertising experiments, focusing on inference for binary outcomes, sample-size/power calculations, and Bayesian reframing.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Onsite
Statistics & Math
5
0
Measuring Conversion Lift from Facebook Ads
Scenario
An advertiser is running a randomized experiment on Facebook. Users are split into:
- Control (unexposed to ads)
- Test (exposed to ads)
For each group you have:
- n_c, x_c = number of users in control, number of conversions in control
- n_t, x_t = number of users in test, number of conversions in test
Assume conversion is binary per user (converted at least once). If only impression-level exposures are available, interpret n as unique users reached, not total impressions.
Question
- How do you estimate the conversion lift and test its statistical significance?
- How large a sample is required to detect a 5% relative lift with 90% power (state assumptions)?
- If a London stakeholder requests a Bayesian approach, how would you re-frame the analysis?
Hints
- Two-proportion z-test or Bayesian posterior for lift; power calculation formula.
Solution
Login required
Show