Calculate A/B sample size, CI, decision rules
Company: Amazon
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Onsite
Design and analyze an A/B test for a signup funnel.
Setup: Baseline conversion p0 = 0.040. Minimum detectable effect (absolute) = 0.004 (i.e., target p1 = 0.044). Two-sided α = 0.05, power = 0.80.
1) Sample size: Compute required sample size per arm for a two-proportion z-test. Show the formula you use and the numeric result.
2) CI and p-value: Suppose after launch you observe Control nC=50,000, xC=2,150 (4.30%) and Treatment nT=49,500, xT=2,090 (4.22%). Compute the p-value for the difference in proportions (both pooled and unpooled variants) and a 95% CI for (pT - pC). Interpret practically.
3) Design choices: Specify primary vs. guardrail metrics and how you would correct for multiple comparisons if you track cancellation rate and latency as guardrails.
4) Power/variance reduction: Explain how you’d use CUPED or stratification to reduce variance and how that changes required sample size.
5) Operational risks: Discuss sequential peeking and how to control type-I error (e.g., alpha-spending). What do you do if traffic is heterogeneous and the global average obscures segment-level reversals?
Quick Answer: This question evaluates skills in experimental design and statistical inference, including sample size calculation, hypothesis testing, confidence interval interpretation, multiple comparison control, variance reduction techniques, and operational considerations for A/B testing.
