A/B Test Sample Size With Variations and Overdispersed Metric Case

You are planning a two‑arm A/B test on a binary conversion metric with:

• Baseline conversion p0 = 8.0% (0.08)
• Relative MDE = 8% (so p1 = 1.08 × p0 = 0.0864; absolute lift Δ = 0.0064)
• Two‑sided α = 0.05, power = 80% (β = 0.20)
• 1:1 allocation (equal sample sizes per arm)

Assume a z‑test for the difference in proportions using a normal approximation.

Tasks

(a) Derive and compute the required per‑arm sample size using the normal approximation to the difference in proportions. Show the z terms you use.

(b) Holding everything else constant (relative to part a), state qualitatively and quantitatively how the required sample size changes when each of the following is changed individually:

1. MDE halves.
2. α is tightened to 0.01 (two‑sided).
3. Power is increased to 90%.
4. Allocation is 75/25 (treatment/control).
5. Observations are clustered at the user level with ICC = 0.02 and average cluster size m = 5 (compute the design effect and apply it).
6. You adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries.

(c) If the primary metric is an overdispersed count (NB2), outline how you would re‑estimate the required sample size.