Compute sample size and test duration
Overview
This question evaluates competency in experimental design and statistical inference for A/B testing, including sample size and power calculations, multiple-comparison correction, hypothesis-test selection (z vs t), confidence-interval interpretation, and randomization/balance diagnostics.
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Statistics & Math
2
0
You will run a two-arm A/B test on a signup funnel. Given: baseline conversion p0 = 4.0%; you care about detecting a 10% relative uplift (p1 = 4.4%); two-sided α = 0.05; power = 80%; Bernoulli outcomes; independent users; simple difference-in-proportions z-test approximation is acceptable.
- Compute the minimum per-variant sample size. Show the exact formula you use and the numeric result.
- Now assume you have 2 primary metrics (conversion and qualified-lead rate). Apply a Bonferroni correction and recompute the per-variant sample size. Explain trade-offs versus using a gatekeeper metric or a hierarchical test.
- Traffic: 1.2M eligible sessions/day, but only 70% meet eligibility checks; you can allocate at most 20% of eligible traffic to the experiment initially. Estimate test duration in days, then propose a safe ramp schedule that preserves statistical validity and operational risk controls.
- Explain when you would use a z-test vs. a t-test here, and interpret a 95% CI that barely excludes zero uplift.
- Describe how you would verify proper randomization and balance across country and platform before looking at outcomes.