Define metrics for high-quality notifications

## Context You are a Data Scientist partnering with a product team at Facebook/Meta that owns push/in-app notifications. The team’s goal is to **send fewer but higher-quality notifications**—i.e., notifications that users find relevant and that improve long-term product outcomes—while avoiding spammy experiences. Assume you can instrument notification events (send, deliver, open/click, dismiss, mute/disable notifications, uninstall), downstream engagement (sessions, time spent, content actions), and longer-term outcomes (retention). You also have user/device attributes and can join events by `user_id` and time. ## Part A — “High-quality” notification definition 1. **What data would you look at** to determine whether notifications are “high-quality” (include both immediate and long-term signals, and consider segmentation)? 2. Propose **one primary success metric** to optimize notification quality, plus **diagnostic metrics** and **guardrails**. Clearly define each metric. ## Part B — Testing a geographic notification feature The team wants to test a new feature: **geographic/nearby-event notifications** (e.g., “A concert is happening near you tonight”). 1. Design an **experiment/measurement plan** to evaluate success (treatment/control definition, unit of randomization, duration). 2. What are the key **threats to validity** (e.g., network effects/interference, novelty, seasonality, selection into location sharing), and how would you mitigate them? 3. What would you conclude and recommend if short-term engagement improves but opt-outs/mutes also increase? Provide your reasoning and any assumptions you need to make.