Compute sample size and plan experiment
Overview
This question evaluates a candidate's proficiency in experimental design and applied statistics, including sample size and power calculations, variance reduction (CUPED), clustering and design-effect adjustments, interim analysis with O'Brien–Fleming alpha spending, multiple-testing control (Benjamini–Hochberg), and causal estimands such as ITT versus CACE. It is commonly asked because interviewers need assurance that a practitioner can translate business treatment goals into a rigorous experiment plan that balances Type I/II error, multiplicity, noncompliance and operational constraints; this falls under the Statistics & Math domain and emphasizes practical application grounded in conceptual understanding.
A/B Test Planning: Paywall Copy Change For New Signups
Context
You are planning an A/B test to evaluate a paywall copy change that targets new signups, with the objective of improving the next-day subscription start rate. The experiment will use equal allocation and two-sided testing.
Given:
- Baseline next-day subscription start rate among new signups: 18%.
- Minimum detectable effect (MDE): +7% relative (target lift to 19.26%).
- Two-sided α = 0.05, power = 0.80, equal allocation.
- Optional CUPED variance reduction using pre-experiment engagement with R² = 0.25.
- If randomizing by household: average household size m = 1.8 (signups per household), ICC = 0.06.
- Up to 4 interim looks with O'Brien–Fleming (OBF) alpha-spending.
- There are 12 secondary metrics; control FDR at 10% using Benjamini–Hochberg (BH).
- Noncompliance: 10% of treated users do not actually see the new copy; 3% of control users are exposed due to caching.
Tasks
- Compute the per-arm sample size ignoring variance reduction and clustering. Show formulas and approximations used.
- With CUPED (R² = 0.25), what is the effective sample size reduction? Recompute the required per-arm sample.
- Adjust for household clustering via the design effect DE = 1 + (m − 1) × ICC. Recompute the per-arm sample size under clustering (with and without CUPED).
- Describe how O'Brien–Fleming boundaries alter Type I error allocation and the practical implications for timeline/power.
- State how you would control FDR at 10% across 12 secondary metrics and interpret discoveries.
- Compute the ITT vs CACE effect given the noncompliance rates (assume monotonicity). How would you report both responsibly to product stakeholders?
Solution
Login required