Design and analyze A/B test with interference
Overview
This question evaluates proficiency in experimental design and causal inference for online A/B testing under interference, covering competencies such as defining estimands and exposure models, handling clustered or dependent data, variance estimation and confidence intervals, power and sample-size calculations, sequential testing, and diagnostics for leakage. It is commonly asked in Analytics & Experimentation interviews because it probes practical application of statistical design and analysis in production-constrained settings, examines understanding of bias introduced by cross-group interference and operational constraints, and falls within the Analytics & Experimentation domain with a primary emphasis on practical application complemented by conceptual understanding.
You must ship a News Feed ranking change where content produced by treated users can be seen by control users, creating interference and within-user correlation across sessions. Only logged-in traffic is eligible. Constraints: max 10% concurrent treatment ramp; analysis window is 14 days; primary metric is sessions per user; guardrails include crash rate and time spent; sequential looks every 2 days are required by policy. Design and analysis questions:
- Choose the experimental unit and randomization scheme to mitigate interference (e.g., graph/ego-network clustering, producer-side randomization, or two-stage randomized encouragement). Justify your choice given a 10% cap and highly skewed degree distribution.
- Define precise estimands: direct effect on consumers, spillover effect, and total effect. Specify exposure conditions and an exposure model that makes your estimands identifiable.
- Describe the estimator and variance: show how you would compute point estimates and 95% CIs with cluster-robust or randomization-inference SEs. State assumptions explicitly and how violations change interpretation.
- Quantify design effect and sample size under clustering: derive the required N given average cluster size m = 120 and ICC ρ = 0.02; show n_eff = n / (1 + (m−1)ρ) and solve for n to achieve power 80% to detect a 0.6% relative lift when baseline mean is 4.0 sessions/user with SD 5.5.
- Diagnostics: propose at least three concrete checks that detect leakage/interference (e.g., cut-edge exposure rates, treated-producer content share in control, placebo effects on non-exposed control users) and how you’d react to each.
- Sequential testing: pick a method (e.g., alpha-spending O’Brien–Fleming or always-valid tests) and outline stopping/decision rules that control Type I error across interim looks.
- If leadership mandates user-level 50/50 randomization (no clustering), propose post-hoc adjustments to bound bias and obtain conservative CIs (e.g., exposure-weighted IV, CUPED with pre-period, sensitivity bounds).