Compute A/B significance, CI, and power

Q: Compute A/B significance, CI, and power

This is a Statistics & Math interview question from CVS Health for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

You run an A/B test for 7 days. Control A: 520 conversions out of 10,000 sessions. Variant B: 630 conversions out of 11,500 sessions. Use a calculator as needed. Assume independent Bernoulli trials. a) Two-proportion z-test for H0: pB − pA = 0 vs two-sided alternative. Compute the pooled-proportion z statistic and the two-sided p-value. b) Compute a 95% confidence interval for (pB − pA) using the unpooled standard error. c) Sample-size planning: with baseline p0 = pA, what equal per-arm sample size n (sessions) is required to detect an absolute lift of 0.7 percentage points (i.e., p1 = p0, p2 = p0 + 0.007) at two-sided α = 0.05 and 80% power using the normal approximation? Use z0.975 = 1.96 and z0.80 = 0.84; report the formula you use and the numeric n rounded up. d) If you were instead running 12 independent metrics, what Bonferroni-corrected per-metric α would you use to maintain family-wise αFWER = 0.05?

Compute A/B significance, CI, and power

Comments (0)