Define metrics for high-quality notifications

Context

You are a Data Scientist partnering with a product team at Facebook/Meta that owns push/in-app notifications.

The team’s goal is to send fewer but higher-quality notifications—i.e., notifications that users find relevant and that improve long-term product outcomes—while avoiding spammy experiences.

Assume you can instrument notification events (send, deliver, open/click, dismiss, mute/disable notifications, uninstall), downstream engagement (sessions, time spent, content actions), and longer-term outcomes (retention). You also have user/device attributes and can join events by user_id and time.

Part A — “High-quality” notification definition

1. What data would you look at to determine whether notifications are “high-quality” (include both immediate and long-term signals, and consider segmentation)?
2. Propose one primary success metric to optimize notification quality, plus diagnostic metrics and guardrails . Clearly define each metric.

Part B — Testing a geographic notification feature

The team wants to test a new feature: geographic/nearby-event notifications (e.g., “A concert is happening near you tonight”).

1. Design an experiment/measurement plan to evaluate success (treatment/control definition, unit of randomization, duration).
2. What are the key threats to validity (e.g., network effects/interference, novelty, seasonality, selection into location sharing), and how would you mitigate them?
3. What would you conclude and recommend if short-term engagement improves but opt-outs/mutes also increase?

Provide your reasoning and any assumptions you need to make.