Determine A/B test sample size drivers
Overview
This question evaluates statistical power and sample size determination for A/B testing, covering hypothesis testing for proportions, derivation using normal approximations, the impact of MDE, α and power choices, allocation ratios, cluster design (ICC and design effect), group‑sequential monitoring, and adjustment for overdispersed count outcomes.
A/B Test Sample Size With Variations and Overdispersed Metric Case
You are planning a two‑arm A/B test on a binary conversion metric with:
- Baseline conversion p0 = 8.0% (0.08)
- Relative MDE = 8% (so p1 = 1.08 × p0 = 0.0864; absolute lift Δ = 0.0064)
- Two‑sided α = 0.05, power = 80% (β = 0.20)
- 1:1 allocation (equal sample sizes per arm)
Assume a z‑test for the difference in proportions using a normal approximation.
Tasks
(a) Derive and compute the required per‑arm sample size using the normal approximation to the difference in proportions. Show the z terms you use.
(b) Holding everything else constant (relative to part a), state qualitatively and quantitatively how the required sample size changes when each of the following is changed individually:
- MDE halves.
- α is tightened to 0.01 (two‑sided).
- Power is increased to 90%.
- Allocation is 75/25 (treatment/control).
- Observations are clustered at the user level with ICC = 0.02 and average cluster size m = 5 (compute the design effect and apply it).
- You adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries.
(c) If the primary metric is an overdispersed count (NB2), outline how you would re‑estimate the required sample size.
Solution
Login required