Explain and validate A/B test assumptions
Overview
This question evaluates proficiency in experimental design, causal inference, and statistical diagnostics for A/B testing, covering competencies such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.
Uber
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
A/B Test Validity: Core Assumptions, Violations, Diagnostics, and Mitigations
You are designing and evaluating an online A/B test for a large, multi-sided consumer product. List the core assumptions required for a valid inference from the experiment and, for each assumption, describe:
- What the assumption formally means.
- One realistic product scenario where it is violated.
- A diagnostic you would run to detect the violation (be specific about tests/visuals).
- A concrete mitigation or redesign (be specific: pre-experiment covariates, pre-period length, which statistical test, whether to use cluster randomization, stratification, CUPED, staggered ramps, etc.).
Cover at least the following assumptions:
- Randomization integrity and sample ratio mismatch (SRM)
- Independence/SUTVA and interference (e.g., network effects, shared inventory)
- Stable unit exposure and cross-over/noncompliance
- Stationarity/time trends and novelty/learning effects
- Metric logging bias/missingness (MCAR/MAR/MNAR)
- Sequential peeking and error inflation
- Heterogeneous treatment effects across key segments
Solution
Login required
Show