Act when A/B result is not significant
Overview
This question evaluates a candidate's competency in experimental design and statistical decision-making, covering sample size calculation, variance-reduction methods, interpretation of non-significant A/B results, multiplicity control, and sequential/Bayesian decision frameworks.
TikTok
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Statistics & Math
0
0
A/B Test Planning and Decision-Making for a 60s Video Change
Context: You are evaluating a product change with completion rate as the primary metric. Baseline completion is 22%. The product team defines the minimum meaningful effect (MME) as a +5% relative lift, i.e., from 22.0% to 23.1% (an absolute +1.1 percentage points). Use a two-sided test with α = 0.05, 80% power, and equal allocation.
Tasks
- Sample size: Compute the required per-arm sample size to detect the MME and explain the formula and assumptions used.
- Variance reduction: Propose variance-reduction methods (e.g., CUPED with pre-period completion, covariate adjustment) and estimate the power gain if R² = 0.25.
- Decision: Given that the current test shows a non-significant lift at α = 0.05, decide whether to continue collecting data, stop, or pivot to a Bayesian, utility-based decision rule.
- Multiplicity: Discuss how you would control multiplicity if multiple related outcomes are monitored.
- Sequential design: Outline a sequential design (e.g., O’Brien–Fleming) that preserves Type I error while allowing early looks.
Solution
Login required
Show