Evaluate new-product notification feature
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
A marketplace team is considering building a feature that notifies buyers when new products relevant to their interests are listed.
How would you determine whether this feature is worth building and launching?
Please address all of the following:
1. **Product hypothesis**
- What user problem is this feature solving?
- Which marketplace outcomes should improve for buyers, sellers, and the platform?
2. **Success metrics**
- Propose leading metrics, primary outcome metrics, and guardrail metrics.
- Consider metrics such as notification delivery rate, open rate, click-through rate, listing views, messages sent, saves, purchases, buyer retention, seller engagement, unsubscribe rate, spam reports, and marketplace liquidity.
- Explain tradeoffs between short-term engagement and long-term notification fatigue.
3. **Experiment design**
- Define the treatment and control.
- Choose a randomization unit and explain why.
- Specify eligibility rules, segmentation, experiment duration, ramp strategy, and how you would estimate sample size / power / minimum detectable effect.
- Discuss risks such as interference, repeated exposure, novelty effects, and seasonality.
4. **Decision framework**
- If purchase labels are delayed or sparse, what proxy metrics would you use?
- How would you decide whether to launch if clicks improve but negative signals such as opt-outs or spam reports also increase?
- What additional analyses would you run before a full rollout?
Quick Answer: This question evaluates a data scientist's competency in product analytics, experimentation design, metric definition, causal inference, and decision frameworks for feature launch, focusing on hypothesis articulation, leading and outcome metrics, trade-offs between short-term engagement and long-term notification fatigue, and handling sparse purchase labels. It is commonly asked to assess the ability to quantify product impact, prioritize features using controlled experiments, reason about proxy and guardrail metrics, and design statistically sound experiments within the Analytics & Experimentation domain; the assessment spans both conceptual understanding and practical application and this is an English summary.