##### Scenario An advertiser is running campaigns on Facebook and wants to know whether the ads increased conversions compared with an unexposed control group. ##### Question Given conversion counts and exposures for test and control groups, how would you estimate the lift and its statistical significance? How large a sample is required to detect a 5% lift at 90% power? If the London stakeholder asks for a Bayesian approach, how would you re-frame the analysis? ##### Hints Two-proportion z-test or Bayesian posterior for lift; power calculation formula.

Meta statistics prompt on Facebook ad conversion lift, covering absolute and relative lift, two-proportion tests, confidence intervals, power for 5% relative lift, and Bayesian beta-binomial reframing.

How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Estimate Lift and Significance in Facebook Ad Campaigns

An advertiser is running campaigns on Facebook and wants to know whether ads increased conversions compared with an unexposed control group.

For each group you have:

n_c , x_c : users and conversions in control.
n_t , x_t : users and conversions in treatment.

Assume conversion is binary per user unless stated otherwise.

Constraints & Assumptions

Define absolute and relative lift.
Use user-level counts, not impression-level rows, unless the analysis explicitly accounts for clustering.
State assumptions for the frequentist test and the sample-size calculation.
Explain a Bayesian reframing for stakeholders who prefer posterior probabilities or credible intervals.

Clarifying Questions to Ask Guidance

Was treatment randomized at user, geo, advertiser, or campaign level?
Is the conversion window fixed and equal across groups?
Are users counted once, or can one user have multiple conversions?
Is the requested 5% lift relative or absolute?
What alpha, baseline conversion rate, and allocation ratio should be assumed?

What a Strong Answer Covers Guidance

Conversion rates p_c = x_c / n_c and p_t = x_t / n_t .
Absolute lift p_t - p_c and relative lift (p_t - p_c) / p_c .
Two-proportion z-test or equivalent GLM, including standard errors and confidence intervals.
Practical concerns for rare events, clustering, multiple conversions, unequal allocation, and sample-ratio mismatch.
Power/sample-size formula for detecting a 5% relative lift, with baseline conversion rate, alpha, power, and allocation stated.
Bayesian framing with beta-binomial posteriors or logistic models, posterior lift distribution, credible intervals, and probability that lift exceeds a business threshold.

Follow-up Questions Guidance

How would the analysis change if randomization was at geo level?
What if treatment has more impressions per user than control?
How would you communicate a non-significant but directionally positive result?
What is the difference between statistical significance and business significance here?

Estimate Lift and Significance in Facebook Ad Campaigns

An advertiser is running campaigns on Facebook and wants to know whether ads increased conversions compared with an unexposed control group.

For each group you have:

n_c , x_c : users and conversions in control.

n_t , x_t : users and conversions in treatment.

Assume conversion is binary per user unless stated otherwise.

Constraints & Assumptions

Define absolute and relative lift.

Use user-level counts, not impression-level rows, unless the analysis explicitly accounts for clustering.

State assumptions for the frequentist test and the sample-size calculation.

Explain a Bayesian reframing for stakeholders who prefer posterior probabilities or credible intervals.

Clarifying Questions to Ask Guidance

Was treatment randomized at user, geo, advertiser, or campaign level?

Is the conversion window fixed and equal across groups?

Are users counted once, or can one user have multiple conversions?

Is the requested 5% lift relative or absolute?

What alpha, baseline conversion rate, and allocation ratio should be assumed?

What a Strong Answer Covers Guidance

Conversion rates p_c = x_c / n_c and p_t = x_t / n_t .

Absolute lift p_t - p_c and relative lift (p_t - p_c) / p_c .

Two-proportion z-test or equivalent GLM, including standard errors and confidence intervals.

Practical concerns for rare events, clustering, multiple conversions, unequal allocation, and sample-ratio mismatch.

Power/sample-size formula for detecting a 5% relative lift, with baseline conversion rate, alpha, power, and allocation stated.

Bayesian framing with beta-binomial posteriors or logistic models, posterior lift distribution, credible intervals, and probability that lift exceeds a business threshold.

Follow-up Questions Guidance

How would the analysis change if randomization was at geo level?

What if treatment has more impressions per user than control?

How would you communicate a non-significant but directionally positive result?

What is the difference between statistical significance and business significance here?

Estimate Lift and Significance in Facebook Ad Campaigns

Quick Overview

Estimate Lift and Significance in Facebook Ad Campaigns