Evaluate a new-listing notification feature
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
A marketplace team is considering a new buyer notification feature that alerts users when newly created listings matching their interests become available. As a data scientist, how would you determine whether this feature is worth building and launching?
Address the following:
1. Define the product objective and the main hypotheses.
2. Propose success metrics and guardrail metrics. Include short-term engagement metrics, downstream marketplace metrics, and possible negative side effects.
3. Explain how you would estimate the opportunity before a full build exists. Consider historical proxy analysis, lightweight MVP or concierge tests, and the risk of selection bias.
4. Design a randomized experiment for an MVP. Specify the unit of randomization, treatment and control, targeting and eligibility rules, experiment duration, and power/MDE considerations.
5. Discuss potential confounding factors such as cannibalization of existing discovery channels, marketplace interference, notification fatigue, and heterogeneous effects across user segments.
6. Explain the final launch decision framework, including both statistical significance and practical business impact.
Quick Answer: This question evaluates product analytics, causal inference, experimentation design, metric definition, and business-impact estimation competencies for a Data Scientist role and falls under the Analytics & Experimentation domain.