Explain and validate A/B test assumptions

Quick Overview

This question evaluates proficiency in experimental design, causal inference, and statistical diagnostics for A/B testing, covering competencies such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.

Explain and validate A/B test assumptions

Company: Uber

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

List the core assumptions required for a valid online A/B test and, for each, describe: (1) what the assumption formally means; (2) one realistic product scenario where it is violated; (3) a diagnostic you would run to detect the violation; and (4) a concrete mitigation/redesign. Cover at least: randomization integrity and sample ratio mismatch (SRM), independence/SUTVA and interference (e.g., network effects, shared inventory), stable unit exposure and cross-over/noncompliance, stationarity/time trends and novelty/learning effects, metric logging bias/missingness (MCAR/MAR/MNAR), sequential peeking and error inflation, and heterogeneous treatment effects across key segments. For each assumption, specify exactly how you would implement the check (e.g., which statistical test or visualization, which pre-experiment covariates, how long a pre-period, whether to use cluster randomization, stratification, CUPED, or staggered ramps).

Quick Answer: This question evaluates proficiency in experimental design, causal inference, and statistical diagnostics for A/B testing, covering competencies such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.

|Home/Analytics & Experimentation/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
13
0

A/B Test Validity: Core Assumptions, Violations, Diagnostics, and Mitigations

You are designing and evaluating an online A/B test for a large, multi-sided consumer product. List the core assumptions required for a valid inference from the experiment and, for each assumption, describe:

  1. What the assumption formally means.
  2. One realistic product scenario where it is violated.
  3. A diagnostic you would run to detect the violation (be specific about tests/visuals).
  4. A concrete mitigation or redesign (be specific: pre-experiment covariates, pre-period length, which statistical test, whether to use cluster randomization, stratification, CUPED, staggered ramps, etc.).

Cover at least the following assumptions:

  • Randomization integrity and sample ratio mismatch (SRM)
  • Independence/SUTVA and interference (e.g., network effects, shared inventory)
  • Stable unit exposure and cross-over/noncompliance
  • Stationarity/time trends and novelty/learning effects
  • Metric logging bias/missingness (MCAR/MAR/MNAR)
  • Sequential peeking and error inflation
  • Heterogeneous treatment effects across key segments
Loading comments...