# Define metrics for high-quality notifications ## Context You are a Data Scientist partnering with a product team at Facebook/Meta that owns push/in-app notifications. The team’s goal is to **send fewer but higher-quality notifications**—i.e., notifications that users find relevant and that improve long-term product outcomes—while avoiding spammy experiences. Assume you can instrument notification events (send, deliver, open/click, dismiss, mute/disable notifications, uninstall), downstream engagement (sessions, time spent, content actions), and longer-term outcomes (retention). You also have user/device attributes and can join events by `user_id` and time. ## Part A — “High-quality” notification definition 1. **What data would you look at** to determine whether notifications are “high-quality” (include both immediate and long-term signals, and consider segmentation)? 2. Propose **one primary success metric** to optimize notification quality, plus **diagnostic metrics** and **guardrails**. Clearly define each metric. ## Part B — Testing a geographic notification feature The team wants to test a new feature: **geographic/nearby-event notifications** (e.g., “A concert is happening near you tonight”). 1. Design an **experiment/measurement plan** to evaluate success (treatment/control definition, unit of randomization, duration). 2. What are the key **threats to validity** (e.g., network effects/interference, novelty, seasonality, selection into location sharing), and how would you mitigate them? 3. What would you conclude and recommend if short-term engagement improves but opt-outs/mutes also increase? Provide your reasoning and any assumptions you need to make. ### Constraints & Assumptions - Preserve the scope, facts, inputs, and requested outputs from the prompt above. - If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it. - Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate. ### Clarifying Questions to Ask - Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric. - State assumptions about instrumentation, randomization, sample size, and data quality. - Separate descriptive analysis from causal claims. ### What a Strong Answer Covers - A metric framework with primary, guardrail, and diagnostic metrics. - A credible analysis or experiment design with clear assumptions and bias checks. - SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant. - An actionable recommendation that explains trade-offs and next steps. ### Follow-up Questions - What sanity checks would you run before trusting the result? - How would you handle novelty effects, seasonality, or selection bias? - What decision would you make if metrics disagree?