Measure Ultrafast Delivery's Impact Using Synthetic Control Method
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Instacart launched Ultrafast Delivery in Miami two months ago and wants to measure its impact on user order volume.
##### Question
Design an approach to estimate the feature’s causal impact on orders. How would you choose an appropriate control geography? Describe the mechanics of your chosen method (e.g., DiD, synthetic control, propensity-score matching). If using a linear mixed-effects model, which variables are fixed versus random?
##### Hints
Cover identification, parallel-trend checks, robustness tests, and metric calculation.
Quick Answer: Instacart causal inference prompt on measuring Ultrafast Delivery impact in Miami, covering synthetic control, control geography selection, difference-in-differences, mixed effects, placebo tests, and order-lift metrics.
Measure Ultrafast Delivery Impact With Synthetic Control
Instacart launched Ultrafast Delivery in Miami two months ago and wants to measure its causal impact on user order volume.
Assume you have daily or weekly panel data for Miami and multiple non-launched geographies, with pre- and post-launch history. You also have covariates such as baseline demand, seasonality, retailer mix, promotions, weather, and customer mix.
Constraints & Assumptions
Miami is the treated geography.
Non-launched geographies form the donor pool.
The goal is causal impact, not just a before/after comparison.
State assumptions and validation checks for each proposed method.
Clarifying Questions to Ask Guidance
What is the outcome: total orders, orders per active user, revenue, retention, or contribution margin?
Did Miami receive any other launches, promotions, or operational changes at the same time?
How much pre-period data is available?
Are nearby geographies potentially affected by spillovers?
What a Strong Answer Covers Guidance
Control geography selection based on pre-launch outcome trends, level, seasonality, retailer mix, demographics, promos, weather, and data quality.
Synthetic control mechanics: choose nonnegative weights on donor geographies to match Miami's pre-period outcomes and covariates, then compare post-period Miami to the weighted synthetic Miami.
Difference-in-differences or event-study as a validation, with parallel trend checks and treatment timing.
Propensity or matching approaches as secondary methods, with limitations.
Linear mixed-effects variant with fixed effects for treatment, post period, treatment-by-post, seasonality, promos, weather, and random intercepts or slopes for geography.
Robustness checks: placebo geographies, leave-one-out donor tests, pre-period fit, sensitivity to donor pool, spillover exclusion, and uncertainty intervals.
Impact metric: absolute order lift, percent lift, cumulative incremental orders, and confidence or placebo-based uncertainty.
Follow-up Questions Guidance
What if synthetic control cannot match Miami well before launch?
Why is raw before/after comparison insufficient?
How would you handle a promotion that launched in Miami at the same time?
What would you report to the business if the effect is positive but uncertain?