Compute A/B significance, CI, and power
Company: CVS Health
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Take-home Project
You run an A/B test for 7 days. Control A: 520 conversions out of 10,000 sessions. Variant B: 630 conversions out of 11,500 sessions. Use a calculator as needed. Assume independent Bernoulli trials. a) Two-proportion z-test for H0: pB − pA = 0 vs two-sided alternative. Compute the pooled-proportion z statistic and the two-sided p-value. b) Compute a 95% confidence interval for (pB − pA) using the unpooled standard error. c) Sample-size planning: with baseline p0 = pA, what equal per-arm sample size n (sessions) is required to detect an absolute lift of 0.7 percentage points (i.e., p1 = p0, p2 = p0 + 0.007) at two-sided α = 0.05 and 80% power using the normal approximation? Use z0.975 = 1.96 and z0.80 = 0.84; report the formula you use and the numeric n rounded up. d) If you were instead running 12 independent metrics, what Bonferroni-corrected per-metric α would you use to maintain family-wise αFWER = 0.05?
Quick Answer: This question evaluates proficiency in statistical inference for A/B testing—specifically two-proportion z-tests, construction of confidence intervals with unpooled standard errors, sample-size and power calculations, and multiple-comparison correction via Bonferroni.