Estimate a non-randomized market launch's effect with difference-in-differences

In a statistics round, the interviewer presents a specific business case and asks you to estimate the effect of a change with difference-in-differences. The report does not describe the case, so this version uses a common setting: a product change was launched to all users in some markets (the treated markets) on a known date, while the remaining markets kept the old experience, and the choice of markets was not random. You have the daily value of an engagement metric for every market for several weeks before and after the launch. How would you estimate the launch's effect on the metric? Explain the estimator, the assumptions it needs and how you would check them, how you would compute its uncertainty, and what you would do if the checks fail. ```hint Two differences A before-and-after comparison in the treated markets mixes the launch with everything else that changed over the same weeks. Ask what the untreated markets can tell you about that. ``` ```hint Use the weeks before the launch The method rests on an assumption about what would have happened without the launch. Think about which part of your data can probe it. ``` ### Clarifying Questions - How were the treated markets chosen, and could the choice be related to their recent metric trends? - Did every treated market launch on the same date, or was the rollout staggered? - Could users in untreated markets be affected by the launch, for example through content or interactions that cross markets? - How many treated and untreated markets are there, and how many weeks of data exist before the launch? - Was the launch announced in advance, so users could react before it took effect? ### What a Strong Answer Covers - The estimator in its two-by-two form and its regression form with market and time fixed effects - The parallel-trends assumption stated precisely, along with no anticipation, no spillovers and stable composition - Pre-launch checks: an event-study plot of pre-launch coefficients and placebo tests - Correct uncertainty: standard errors clustered by market, and remedies when there are few markets - Fallbacks when the checks fail, such as better-matched comparison markets, synthetic control, or estimators built for staggered timing - Interpretation of the estimate as an average effect on the treated markets ### Follow-up Questions - If the launch was staggered across markets over several weeks, why can the standard two-way fixed-effects regression mislead, and what would you use instead? - The pre-launch trends diverge slightly. What can you still conclude? - There is only one treated market. How do you estimate the effect and its uncertainty? - How would you decide whether to model the metric in levels or in logs?