Measure and mitigate notification spam

Facebook sends many notification types (e.g., friends' posts, comments, birthdays, events). Design a rigorous measurement plan to determine whether our notification system is healthy or 'spammy' and what to ship next. Be specific: 1) Precisely define primary success metrics (e.g., per-notification CTR, 1-day and 7-day post-click actions such as commenting/RSVP/purchase), guardrails (mute/opt-out/disable rates, hide/report rate, notification-to-session conversion, retention impact), and counterfactual-aware long-term metrics (28-day retention, notification-driven DAU lift vs substitution). Provide exact metric formulas and event windows. 2) Propose an experimental design that captures cross-notification interference and fatigue: e.g., portfolio-level randomized throttling, user-level long-term holdouts, and type-level factorials. Explain unit of randomization, power trade-offs, and how you'll attribute effects when multiple notifications arrive in a session. 3) Decide how to optimize when 'overall ecosystem' metrics improve but a distinct cohort experiences harm. Specify a decision rule (e.g., weighted objective or two-stage gating) that protects users with negative experience. Include concrete thresholds for halting (e.g., if 7-day opt-out rate for the bottom decile cohort exceeds X bp with 95% CI). 4) If we allow easy opt-outs, detail how you'll prevent silent user loss: instrumentation to detect pre-opt-out frustration, early-warning leading indicators, and re-engagement safeguards. Describe how you'll correct for survivorship/selection bias from opt-outs when estimating long-term effects. 5) Outline a monitoring plan (dashboards and anomaly alerts) and a rollout/ramp policy that limits risk while gathering evidence. Include how you'll back-test decisions and sunset harmful notification types.