Compute sample size and test duration
Company: Meta
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Onsite
You will run a two-arm A/B test on a signup funnel.
Given: baseline conversion p0 = 4.0%; you care about detecting a 10% relative uplift (p1 = 4.4%); two-sided α = 0.05; power = 80%; Bernoulli outcomes; independent users; simple difference-in-proportions z-test approximation is acceptable.
- Compute the minimum per-variant sample size. Show the exact formula you use and the numeric result.
- Now assume you have 2 primary metrics (conversion and qualified-lead rate). Apply a Bonferroni correction and recompute the per-variant sample size. Explain trade-offs versus using a gatekeeper metric or a hierarchical test.
- Traffic: 1.2M eligible sessions/day, but only 70% meet eligibility checks; you can allocate at most 20% of eligible traffic to the experiment initially. Estimate test duration in days, then propose a safe ramp schedule that preserves statistical validity and operational risk controls.
- Explain when you would use a z-test vs. a t-test here, and interpret a 95% CI that barely excludes zero uplift.
- Describe how you would verify proper randomization and balance across country and platform before looking at outcomes.
Overview: This question evaluates competency in experimental design and statistical inference for A/B testing, including sample size and power calculations, multiple-comparison correction, hypothesis-test selection (z vs t), confidence-interval interpretation, and randomization/balance diagnostics.
Community answers
Answer by SS
(1) Sample size:
Δ = 0.044 − 0.04 = 0.004
Variance: 0.04⋅0.96+0.044⋅0.956=0.0384+0.042064=0.0804640.04 \cdot 0.96 + 0.044 \cdot 0.956 = 0.0384 + 0.042064 = 0.0804640.04⋅0.96+0.044⋅0.956=0.0384+0.042064=0.080464
Multiply by 7.84:
n=7.84⋅0.0804640.0042=7.84⋅0.0804640.000016=7.84⋅5030.25≈39,455n = 7.84 \cdot \frac{0.080464}{0.004^2} = 7.84 \cdot \frac{0.080464}{0.000016} = 7.84 \cdot 5030.25 \approx 39,455n=7.84⋅0.00420.080464=7.84⋅0.0000160.080464=7.84⋅5030.25≈39,455
Answer by SS
(2) ### Step 1: Adjust α for Bonferroni
Original α = 0.05 (two-sided)
Two metrics → α_per_test = 0.05 / 2 = 0.025
Step 2: Update z for two-sided test
z_{1−α/2} = z_{1−0.025/2} = z_{0.9875} ≈ 2.24 (instead of 1.96)
Power z_{1−β} = 0.84 (same)
Sum: 2.24 + 0.84 = 3.08
Square: 3.08² ≈ 9.49
Step 3: Variance and Δ
Variance = p₀(1−p₀) + p₁(1−p₁) ≈ 0.080464
Δ = 0.004
Step 4: Sample size
n=9.49⋅0.0804640.0042=0.7640.000016≈47,750n = \frac{9.49 \cdot 0.080464}{0.004^2} = \frac{0.764}{0.000016} \approx 47,750n=0.00429.49⋅0.080464=0.0000160.764≈47,750
✅ Per-variant sample size with Bonferroni (2 metrics) ≈ 47,800 users per group
Answer by SS
(3)## Step 1: Compute daily experiment traffic
Total sessions/day = 1.2M
Eligible fraction = 70% → eligible sessions/day = 1.2M × 0.7 = 840,000/day
Max allocation to experiment initially = 20% → experimental traffic/day = 840,000 × 0.2 = 168,000/day
Step 2: Required sample size per group
From the previous calculation:
n_per_variant ≈ 39,400 (for a single primary metric, difference in conversion 0.04 → 0.044)
Two variants → total sample required = 2 × 39,400 ≈ 78,800 users
Note: If using Bonferroni for 2 primary metrics, n_per_variant ≈ 47,800 → total ≈ 95,600
Step 3: Estimate test duration
Duration = required sample / daily experiment traffic
Single metric case:
Days=78,800168,000≈0.47 days≈∗∗1day∗∗(roundup)\text{Days} = \frac{78,800}{168,000} \approx 0.47 \text{ days} \approx 1 day (round up)Days=168,00078,800≈0.47 days≈∗∗1day∗∗(roundup)
Two-metric Bonferroni case:
Days=95,600168,000≈0.57 days≈∗∗1day∗∗\text{Days} = \frac{95,600}{168,000} \approx 0.57 \text{ days} \approx 1 dayDays=168,00095,600≈0.57 days≈∗∗1day∗∗
✅ So the experiment only needs ~1 day of full allocated traffic to reach the minimum sample size.
Note: Often we run slightly longer to account for session randomness, holiday/weekend effects, or data quality, e.g., 2–3 days.