Compute A/B sample size under clustering

Q: Compute A/B sample size under clustering

This question evaluates competency in experimental design and statistical inference for A/B testing, specifically sample size calculation for two-sample proportions with unequal allocation, clustering effects (ICC and design effect), and adjustments for bot removal and data-quality attrition.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Statistics & Math question, commonly asked during Technical Screen rounds at Uber.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Question

A/B Test Sample Size With Unequal Allocation, Clustering, and Attrition

Context

You are planning a two-arm signup A/B test (binary outcome: convert vs. not) with a 2:1 traffic split (control:treatment). You need the analyzable and gross (pre-filter) sample size per variant and the expected test duration, accounting for clustering by user, bot removal, and data-quality attrition.

Given

Baseline conversion (control) p0 = 6% = 0.06
Target relative uplift = +5% ⇒ MDE (absolute) Δ = 0.05 × p0 = 0.003 ⇒ treatment p1 = 0.063
Two-sided α = 0.05, power = 0.80
Allocation: control:treatment = 2:1 ⇒ λ = n_T / n_C = 0.5
Sessions cluster by user: mean sessions/user m = 1.4, ICC ρ = 0.03
15% sessions are bots removed post-hoc
8% expected attrition from data-quality filters
Traffic capacity: 120,000 sessions/day (total)

Task

Provide the formulas for two-sample tests of proportions with unequal allocation.
Compute the required analyzable sample size per variant (control, treatment).
Apply the design effect for clustering.
Inflate to gross (pre-filter) sessions accounting for 15% bots and 8% attrition.
Convert the result to an estimated test duration given 120,000 sessions/day under a 2:1 split.
State assumptions and how violations (variance mis-specification, sequential peeking) would change the plan.

Compute A/B sample size under clustering

Quick Overview

Compute A/B sample size under clustering

A/B Test Sample Size With Unequal Allocation, Clustering, and Attrition

Context

Given

Task

Write your answer