Explain and validate A/B test assumptions
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
List the core assumptions required for a valid online A/B test and, for each, describe: (1) what the assumption formally means; (2) one realistic product scenario where it is violated; (3) a diagnostic you would run to detect the violation; and (4) a concrete mitigation/redesign. Cover at least: randomization integrity and sample ratio mismatch (SRM), independence/SUTVA and interference (e.g., network effects, shared inventory), stable unit exposure and cross-over/noncompliance, stationarity/time trends and novelty/learning effects, metric logging bias/missingness (MCAR/MAR/MNAR), sequential peeking and error inflation, and heterogeneous treatment effects across key segments. For each assumption, specify exactly how you would implement the check (e.g., which statistical test or visualization, which pre-experiment covariates, how long a pre-period, whether to use cluster randomization, stratification, CUPED, or staggered ramps).
Quick Answer: This question evaluates proficiency in experimental design, causal inference, and statistical diagnostics for A/B testing, covering competencies such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.