Design a causal evaluation without A/B testing
Quick Overview
This question evaluates a data scientist's competency in causal inference, time-series analysis, and synthetic control methodology for non-randomized experiments, including donor pool construction, pre/post diagnostics, and sensitivity and heterogeneity assessments.
Non-Randomized Launch Decision via Synthetic Control: Complete Analysis Plan
You need to make a go/no-go decision for a high-impact feature that cannot be A/B tested due to policy/infrastructure constraints. Assume we can gate the feature to one or a small number of units (e.g., a geography × platform cell) and measure time-series KPIs across comparable units.
Propose a complete analysis plan using Synthetic Control (or justify an alternative if SCM is unsuitable). Address the following:
(a) Treatment unit and donor pool construction
- Define the treatment unit precisely.
- Specify how you will build the donor pool.
- Eligibility/exclusion rules to prevent contamination, spillovers, and anticipation effects.
(b) Pre-intervention window and external factors
- Pre-period length and rationale.
- How you will handle seasonality, holidays, macro shocks, and data latency/backfill.
(c) Metrics and decision rubric
- Primary outcome and guardrail metrics.
- Pre-registered success thresholds and a clear go/no-go decision rule.
(d) Predictors, weights, and tuning
- Which predictors to include (outcome lags vs. covariates).
- Weight constraints and how you will tune hyperparameters.
(e) Diagnostics and inference
- Required diagnostics before trusting effects (e.g., pre-period RMSPE targets, placebo tests in-space/in-time, pre/post fit plots).
- Inference approach (MSPE ratio/permutation tests, uncertainty bands for pointwise and cumulative effects).
(f) Sensitivity and spillovers
- Sensitivity analyses (leave-one-out donors, alternative windows, augmented/regularized SCM, donor reweighting).
- How you will bound and assess spillovers/contamination.
(g) Heterogeneity and persistence
- Subgroup and dynamic-effect analyses to assess heterogeneity and persistence/decay.
(h) Fallbacks
- What you will do if pre-period fit is poor or donors are scarce.
(i) Launch mapping
- How results translate into a staged launch plan, rollback criteria, and post-launch monitoring.