Compute A/B sample size under clustering
Overview
This question evaluates competency in experimental design and statistical inference for A/B testing, specifically sample size calculation for two-sample proportions with unequal allocation, clustering effects (ICC and design effect), and adjustments for bot removal and data-quality attrition.
Uber
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Statistics & Math
0
0
A/B Test Sample Size With Unequal Allocation, Clustering, and Attrition
Context
You are planning a two-arm signup A/B test (binary outcome: convert vs. not) with a 2:1 traffic split (control:treatment). You need the analyzable and gross (pre-filter) sample size per variant and the expected test duration, accounting for clustering by user, bot removal, and data-quality attrition.
Given
- Baseline conversion (control) p0 = 6% = 0.06
- Target relative uplift = +5% ⇒ MDE (absolute) Δ = 0.05 × p0 = 0.003 ⇒ treatment p1 = 0.063
- Two-sided α = 0.05, power = 0.80
- Allocation: control:treatment = 2:1 ⇒ λ = n_T / n_C = 0.5
- Sessions cluster by user: mean sessions/user m = 1.4, ICC ρ = 0.03
- 15% sessions are bots removed post-hoc
- 8% expected attrition from data-quality filters
- Traffic capacity: 120,000 sessions/day (total)
Task
- Provide the formulas for two-sample tests of proportions with unequal allocation.
- Compute the required analyzable sample size per variant (control, treatment).
- Apply the design effect for clustering.
- Inflate to gross (pre-filter) sessions accounting for 15% bots and 8% attrition.
- Convert the result to an estimated test duration given 120,000 sessions/day under a 2:1 split.
- State assumptions and how violations (variance mis-specification, sequential peeking) would change the plan.
Solution
Login required
Show