Measure Ultrafast Delivery Impact With Synthetic Control
Instacart launched Ultrafast Delivery in Miami two months ago and wants to measure its causal impact on user order volume.
Assume you have daily or weekly panel data for Miami and multiple non-launched geographies, with pre- and post-launch history. You also have covariates such as baseline demand, seasonality, retailer mix, promotions, weather, and customer mix.
Constraints & Assumptions
-
Miami is the treated geography.
-
Non-launched geographies form the donor pool.
-
The goal is causal impact, not just a before/after comparison.
-
State assumptions and validation checks for each proposed method.
Clarifying Questions to Ask
-
What is the outcome: total orders, orders per active user, revenue, retention, or contribution margin?
-
Did Miami receive any other launches, promotions, or operational changes at the same time?
-
How much pre-period data is available?
-
Are nearby geographies potentially affected by spillovers?
What a Strong Answer Covers
-
Control geography selection based on pre-launch outcome trends, level, seasonality, retailer mix, demographics, promos, weather, and data quality.
-
Synthetic control mechanics: choose nonnegative weights on donor geographies to match Miami's pre-period outcomes and covariates, then compare post-period Miami to the weighted synthetic Miami.
-
Difference-in-differences or event-study as a validation, with parallel trend checks and treatment timing.
-
Propensity or matching approaches as secondary methods, with limitations.
-
Linear mixed-effects variant with fixed effects for treatment, post period, treatment-by-post, seasonality, promos, weather, and random intercepts or slopes for geography.
-
Robustness checks: placebo geographies, leave-one-out donor tests, pre-period fit, sensitivity to donor pool, spillover exclusion, and uncertainty intervals.
-
Impact metric: absolute order lift, percent lift, cumulative incremental orders, and confidence or placebo-based uncertainty.
Follow-up Questions
-
What if synthetic control cannot match Miami well before launch?
-
Why is raw before/after comparison insufficient?
-
How would you handle a promotion that launched in Miami at the same time?
-
What would you report to the business if the effect is positive but uncertain?