Compute CIs, power, and multiple testing
Company: Amazon
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Onsite
A baseline conversion rate is p0 = 0.051 with N0 = 100,000 users. You plan to detect a +5% relative lift (p1 = 1.05 × p0) at α = 0.05 (two-sided) with 80% power.
Tasks:
1) Compute a 95% Wald (and an Agresti–Coull or Wilson) confidence interval for p0. Explain why Wilson/AC may be preferable.
2) Approximate the per-variant sample size required for 80% power to detect the target lift using a normal approximation for two proportions. State all formulas and assumptions.
3) If you simultaneously test three metrics (conversion, AOV, retention), apply a Bonferroni or Holm correction and provide the adjusted α for each. Discuss trade-offs vs FDR control (Benjamini–Hochberg).
4) Your data exhibits user-level clustering (repeat visitors). Explain why independence is violated and how to correct standard errors (e.g., cluster-robust SEs or user-level aggregation) and how that affects power.
Quick Answer: This question evaluates a candidate's competence in statistical inference for binary outcomes, confidence-interval methods, power and sample-size estimation, multiple-testing corrections, and handling clustered user-level data.
