Analyze Key Metrics for Notification System Success
Overview
This question evaluates a data scientist's ability to select and interpret key product metrics, reason about trade-offs between short-term engagement and medium-term retention, and analyze randomized A/B experiment outcomes to detect user annoyance or churn.
Scenario
You are evaluating a new push-notification system for a social app. The goal is to determine whether the new system improves user value without increasing user annoyance or churn.
Assume you can run a randomized A/B experiment (50/50 at the user level), collect notification- and user-level events, and observe metrics for at least 2–4 weeks. Consider both short-term engagement and medium-term retention outcomes.
Questions
- Define the key metrics you would track to measure notification quality.
- How would you decide whether to launch the new notification experience?
- If one metric improves while another declines after the experiment, how would you analyze and reconcile the discrepancy?
Hints: Consider engagement, retention, churn, user annoyance; design an A/B test and deep-dive segment analysis.
Solution
Login required