Explain power drivers and resolve unexpected A/B results
Overview
This question evaluates a data scientist's mastery of A/B testing fundamentals — statistical power and sample-size calculations, effect-size and variance considerations, allocation strategies, variance-reduction methods such as CUPED, and experiment diagnostics including SRM, instrumentation audits, imbalance and segmentation checks.
A/B Testing: Power, Sample Size, Allocation, and Diagnostics
You are analyzing a two-proportion (binary conversion) A/B test with independent users, no clustering/spillover, and equal exposure eligibility per day unless specified. Answer all parts concisely and show calculations where requested.
(a) Define Power and Rank the Levers
Define statistical power for a two-proportion A/B test and list the primary levers that increase power, ranking them by typical practical impact (largest to smallest). Briefly explain trade-offs. Include:
- Effect size (MDE)
- Variance/metric volatility
- Sample size
- Allocation ratio
- Alpha
- Variance-reduction (e.g., CUPED)
- Bucketing/stratification
- Test duration/seasonality
(b) Baseline Scenario: Sample Size and Duration
Given:
- Baseline conversion p0 = 5.00%
- Target relative lift = +7% ⇒ p1 = 5.35% (Δ = 0.35 pp)
- Two-sided alpha = 0.05
- Desired power = 0.80
- Equal allocation (50/50)
- 80,000 eligible users per day
- 10% post-randomization attrition (i.e., only 90% produce analyzable outcomes)
Compute:
- Required sample size per variant (analyzable users)
- Minimum test duration (days)
Show formulas and numeric results.
(c) With CUPED (R² = 0.30)
Recompute (b) assuming CUPED achieves a 30% relative variance reduction (R² = 0.30). What is the new sample size per variant and duration?
(d) Unequal Allocation 90/10
How does switching to a 90/10 allocation (90% control, 10% treatment) affect power at fixed total traffic? Provide intuition and, if possible, a quantitative comparison to equal split.
(e) Negative Significant Result: Diagnostic Plan
Your test, run for the duration from (b), returns a statistically significant −2% lift (treatment worse), contrary to your prior expectation of +7%. Outline a step-by-step diagnostic plan before drawing conclusions. Include:
- SRM checks (and why)
- Instrumentation/metric definition audits
- Bot/geo/device imbalance
- Novelty/learning effects
- Outlier clipping
- Simpson’s paradox via key segments
- Guardrail metrics
- Peeking/stopping risk
(f) Decision Tree After Diagnostics
Propose an evidence-based decision tree for what to do next. Specify when to:
- (i) Ship
- (ii) Iterate with a follow-up test (name one concrete design change to test)
- (iii) Rerun (state the precise condition that justifies a rerun)
Solution
Login required