Estimate a Launch Impact with Difference in Differences
Company: Pinterest
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
A feature launches in a set of treated regions while comparable control regions do not receive it. You observe an outcome before and after launch and cannot randomize retrospectively. Explain how you would estimate the effect using difference in differences, validate its assumptions, and communicate the limits of the result.
### Constraints & Assumptions
- Treatment starts at a known time, and treatment assignment is defined at the region level.
- Repeated observations are available for multiple pre- and post-launch periods.
- Other events may affect regions differently, so the design must be diagnosed rather than assumed valid.
### Clarifying Questions to Ask
- Why were treated regions selected, and did selection depend on recent outcome trends?
- Did every treated region launch at the same time and with the same intensity?
- Are control regions exposed to spillovers, anticipation, or competing interventions?
- What is the outcome grain, and what decision does the effect estimate support?
- How many region clusters are available for inference?
### What a Strong Answer Covers
- The two-by-two difference and a regression formulation with treatment-group and post-period interaction.
- A clear interpretation of the interaction coefficient as an average treatment effect under identifying assumptions.
- Evidence on pre-treatment trends through plots, event-study leads, and appropriate placebo checks.
- Region and time fixed effects when using panel data, with uncertainty clustered at the treatment-assignment level.
- Threats from selection, concurrent shocks, spillovers, composition changes, anticipation, and staggered adoption.
- Sensitivity analyses and restrained causal language when assumptions remain uncertain.
### Follow-up Questions
1. Does a nonsignificant pretrend test prove parallel trends?
2. How would you handle treatment starting at different times across regions?
3. What if only a handful of regions are treated?
4. How can a change in user composition bias the outcome?
Quick Answer: Estimate a regional product launch effect with difference in differences when retrospective randomization is unavailable. Define the estimand, test parallel-trend credibility, use event-study and placebo diagnostics, cluster uncertainty correctly, and handle spillovers, selection, and staggered adoption.