Estimate Causal Impact Using Synthetic Control Methods
Overview
This question evaluates a data scientist's competency in causal inference and observational impact estimation within the Analytics & Experimentation domain, emphasizing handling time-series seasonality, cross-sectional heterogeneity, panel and exposure data, and identification challenges.
Estimating Causal Impact After a 100% Rollout (No Holdout)
Context
A product feature was launched to 100% of traffic simultaneously, so there is no explicit control or holdout group. You need to estimate the causal impact of the launch on a key business metric (e.g., conversion rate, bookings, revenue), in a marketplace with strong seasonality and heterogeneity across geographies, devices, and user segments.
Assume you have historical data (pre- and post-launch), unit-level and geo-level panels, exposure logs (who actually saw/used the feature), and standard product/marketing covariates.
Task
Outline:
- Possible causal inference methodologies suitable for this setting.
- The data each method requires.
- Key identification assumptions and how you would validate them.
- How you would quantify and communicate uncertainty.
Hints: Discuss pre–post analysis, synthetic controls, difference-in-differences, and propensity-based approaches; emphasize validation and sensitivity checks.
Solution
Login required