Choose tests for rare-event A/B analysis
Company: HBO
Role: Data Scientist
Category: Statistics & Math
Difficulty: hard
Interview Round: Take-home Project
You observe binary cancellation outcomes in an A/B test with very low event rates. Baseline weekly cancellation p0 = 0.003; you want to detect p1 = 0.0035 with alpha = 0.05 (two-sided) and power = 0.80 using equal allocation. (1) Decide whether to use a normal approximation z-test, Fisher's exact test, or a mid-p variant, and justify based on expected counts; (2) Compute/outline the sample size per arm for the z-test with and without a continuity correction; (3) Show how CUPED with a continuous covariate (pre-period watch-hours) changes the variance and effective sample size—derive the adjustment using 1−R^2; (4) Provide an exact or conservative confidence interval for the risk difference and for the relative risk; (5) If you perform weekly interim looks (4 total), specify an alpha-spending function and approximate adjusted critical values; (6) Explain when a Bayesian Beta–Binomial model would be preferable, and how you would set priors and a stopping rule (e.g., P(Δ<0) > 0.95).
Quick Answer: This question evaluates expertise in statistical experiment design for rare binary outcomes, covering hypothesis test selection for low-event-rate proportions, sample-size calculation with and without continuity correction, variance reduction via CUPED, confidence-interval construction for risk difference and relative risk, interim alpha spending, and Bayesian Beta–Binomial modeling; it falls under the Statistics & Math domain with emphasis on A/B testing and experimental design. It is commonly asked to assess how a candidate balances statistical validity, power, and sequential monitoring in low-probability A/B tests and requires both conceptual understanding of inference principles and practical application skills such as analytic sample-size derivation and sequential decision criteria.