Determine Sample Size for Promotion Campaign A/B Test

##### Scenario Uber plans to launch a promotion campaign and wants to evaluate its effectiveness with an A/B experiment. Two concrete variants of the offer have come up in interviews: - A spend-threshold discount (e.g., **20% off when a user spends $40**), and - A geo-targeted promo limited to a single market (e.g., a **January-2024 campaign available only to San-Francisco users**). Treat the offer as a generic promotion; the reasoning below applies to either framing. ##### Question 1. **Business rationale.** Why might the company want to launch this promotion campaign? (If geo-limited, why restrict it to a single market like San Francisco?) 2. **Success metrics.** Which business metrics would you monitor to judge success? Distinguish primary, secondary/diagnostic, and guardrail metrics. 3. **Sample size.** How would you determine the required sample size for the study? What inputs, assumptions, and factors influence it? 4. **Interpreting the MDE.** Revenue is chosen as the primary metric and the product manager sets the minimum detectable effect (MDE) to **0.5%**. How do you interpret this value and how does it affect the sample-size calculation? 5. **Non-significant result.** Suppose the observed lift is **+0.3%** but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager? ##### Hints Think about goal alignment, primary/secondary/guardrail KPIs, baseline mean and variance, power and Type-I/Type-II errors, effect size, significance level, variance reduction (e.g., CUPED), and how to explain a small, directional-but-non-significant effect to a stakeholder.