Evaluate impact without randomized experiments
Overview
This question evaluates a data scientist's competence in causal inference and identification strategies for estimating promotion effects from observational longitudinal data, emphasizing challenges such as staggered rollout, interference (spillovers), seasonality, and uncertainty quantification.
Estimating a Promotion's Causal Effect Without an Experiment
Context
You need to estimate the causal impact of a marketing promotion on engagement (e.g., trips, bookings, revenue) when a randomized experiment is not feasible. You have longitudinal data at user/geo × time granularity, with information on promotion eligibility/exposure, outcomes, covariates, and rollout timing that may be staggered across units.
Task
Propose and justify at least three distinct identification strategies to estimate the promotion’s causal effect. For each strategy, clearly state:
- Identification setup and estimator(s)
- Key assumptions
- Diagnostics and robustness checks
- Failure modes and how you would detect/mitigate them
You may choose from (or add to) the following:
- Difference-in-Differences (DiD) / Event Study with staggered rollout using modern estimators (avoid naive TWFE bias), including parallel-trends checks and placebo tests.
- Synthetic Control at the geo level with pre-period fit metrics and sensitivity to donor pool/regularization.
- Instrumental Variables exploiting exogenous variation (e.g., eligibility rules, random delivery outages), with relevance/exclusion checks and weak-IV diagnostics.
- Regression Discontinuity if eligibility thresholds exist (e.g., tenure ≥ 30 days), with bandwidth/functional-form sensitivity and McCrary density tests.
- Propensity-score or doubly-robust learners (IPW/AIPW, causal forests) with overlap checks and covariate balance.
Additionally, explain how you would:
- Report ATE/ATT with uncertainty
- Account for interference/network effects (spillovers)
- Control for seasonality and concurrent campaigns
Solution
Login required