How to evaluate new listing notifications?
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You work on an online marketplace. The product team wants to build a feature that notifies buyers when new products are listed that match their interests.
How would you determine whether this feature is worth building and launching?
Please discuss:
1. The business and product objective you would optimize for.
2. The primary, secondary, and guardrail metrics you would track.
3. How you would design an experiment to measure the causal impact of the feature.
4. How you would handle common pitfalls such as selection bias, notification fatigue, and marketplace interference where multiple buyers may compete for the same limited inventory.
Your answer should consider both buyer-side outcomes and seller-side marketplace outcomes, and should explain why some metrics such as open rate or click-through rate may be insufficient on their own.
Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, metric selection (primary, secondary, guardrail), and marketplace-aware analytics for buyer notification features.