Quasi-Experimental Analysis for Product Decisions
Asked of: Data Scientist
Last updated
-
What it is Quasi-experimental analysis estimates the causal impact of a product change when randomized A/B testing isn’t feasible. It leverages natural experiments or structured observational data—using designs like difference‑in‑differences, regression discontinuity, synthetic control, or interrupted time series—to approximate the counterfactual.
-
Why interviewers ask about it Many high-impact decisions at large platforms (e.g., ranking changes, policy launches, pricing, growth notifications) can’t be cleanly randomized or risk harmful spillovers. Interviewers want to see if you can pick an identification strategy, state assumptions, stress-test them, and deliver a decision-ready estimate under real product constraints.
-
Core ideas to know
- Identification toolbox: DiD, event studies, regression discontinuity, synthetic control, instrumental variables, and interrupted time series/BSTS.
- Assumptions matter: parallel trends (DiD), continuity/no manipulation at cutoff (RDD), unaffected controls (synthetic/ITS), no anticipation, limited spillovers.
- Design first: define estimand (ATE/ATT), units, timing, exposure; prevent contamination and interference; pre-register metrics and guardrails.
- Diagnostics: pre-trend checks, placebo tests, falsification outcomes, sensitivity to control sets and bandwidths, negative controls.
- Estimation details: cluster-robust errors, seasonality/holiday controls, staggered adoption estimators, small-sample corrections.
- Practicalities: choose comparable controls (matching, synthetic donors), monitor integrity (backfills, outages), and validate metric stability.
- Communication: quantify uncertainty, discuss assumption plausibility, highlight heterogeneity and business implications.
-
A common pitfall Candidates jump straight to DiD or CausalImpact without verifying assumptions or product mechanics. For example, they ignore obvious pre-trend breaks from seasonal shocks (Black Friday), use “controls” partially affected by the launch (geo spillovers), or let thresholds be gameable in RDD. They also under-specify the estimand (what effect, on whom, and when), making results hard to interpret or act on.
-
Further reading
- The State of Applied Econometrics: Causality and Policy Evaluation (Athey & Imbens, JEP) — clear overview of DiD, RDD, and synthetic control, with guidance on assumptions and interpretation. (aeaweb.org)
- CausalImpact (Google) — practical interrupted time series/Bayesian structural time series with discussion of required assumptions and control series selection. (google.github.io)
- Round 2: A Survey of Causal Inference Applications at Netflix — industry case studies showing when quasi-experiments are used instead of A/B tests and the challenges teams face. (engineering.fyi)
Related concepts
- Causal Inference And Quasi-Experiments
- Statistical Inference For Experiments
- A/B Testing And Causal InferenceAnalytics & Experimentation
- A/B Testing And Experiment DesignAnalytics & Experimentation
- A/B Testing And Experiment Analysis
- Difference-In-Differences And Quasi-ExperimentsAnalytics & Experimentation