Choose alternatives when randomization fails
Overview
This question evaluates proficiency in causal inference and experimentation analytics for a Data Scientist role, focusing on selecting appropriate identification strategies, designing placebo and pre‑trend tests, and specifying robustness and sensitivity analyses for non‑randomized feature adoption.
Causal Impact of an Autoloaded Feature Without Clean Randomization
Context
You need to estimate the causal effect of a new autoloaded feature that is only enabled for opted‑in power users. Because eligibility is restricted and adoption is not randomized, standard A/B testing is not feasible. You have user‑level panel data (pre/post outcomes, eligibility, adoption timing, and rich covariates like app version, device, region, prior engagement).
Tasks
(a) Choose one approach among Difference‑in‑Differences (DiD), Synthetic Control, Instrumental Variables (IV), or Propensity Score Matching (PSM). Justify the choice and clearly state its identifying assumptions.
(b) Design placebo and pre‑trend tests and specify robustness checks (e.g., varying windows, alternative control groups/covariates).
(c) Propose sensitivity analyses (e.g., negative controls, Rosenbaum bounds or related confounding robustness analyses).
(d) Explain the likely directional bias risks relative to a randomized A/B test, and why.
Solution
Login required