Measure Local News Launch Success
Company: Nextdoor
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Nextdoor is considering launching a Local News feature that recommends neighborhood-specific news articles inside the app. The goal is to increase user value by making the product more locally relevant, but there are two important complications: users in the same neighborhood may influence one another, and the supply and quality of local news may vary a lot across geographies.
As the data scientist for this launch, answer the following:
1. How would you define success for the feature? Propose a metric framework that includes:
- one primary success metric,
- supporting funnel metrics,
- guardrail metrics.
2. How would you design an experiment to estimate the causal impact of the feature? Be specific about:
- the randomization unit,
- the target population,
- experiment duration,
- how you would handle pre-existing differences across neighborhoods,
- how you would interpret short-term engagement versus long-term retention.
3. Compare user-level A/B testing and geography-level A/B testing, such as randomizing by neighborhood or city. What are the main pros and cons of each approach in this setting? Under what conditions would you choose one over the other?
4. How would your plan change if treatment effects can spill over across nearby users, or if some neighborhoods have very little local news inventory?
Your answer should discuss metric tradeoffs, interference, selection bias, and practical experimentation constraints.
Quick Answer: This question evaluates experimentation design, causal inference, metric definition, and the handling of interference and selection bias in product analytics.