Determine Metrics to Evaluate Notification Impact on Users
Quick Overview
Evaluates product analytics judgment for Facebook push notifications, including how to measure helpful versus harmful notifications, design an event-notification experiment, and balance short-term clicks against retention, fatigue, and opt-out guardrails. Strong answers build a metric hierarchy and use causal measurement.
Determine Metrics to Evaluate Notification Impact on Users
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Facebook has multiple types of push notifications and is considering launching a new one that tells you a friend plans to attend an event.
##### Question
Which data and metrics would you use to determine whether an existing notification is helpful or harmful to users? What analyses or experiments would you run to decide whether to launch the new event-based notification?
##### Hints
Think funnel metrics, user experience trade-offs, A/B tests, downstream engagement and churn.
Quick Answer: Evaluates product analytics judgment for Facebook push notifications, including how to measure helpful versus harmful notifications, design an event-notification experiment, and balance short-term clicks against retention, fatigue, and opt-out guardrails. Strong answers build a metric hierarchy and use causal measurement.
Determine Metrics to Evaluate Notification Impact on Users
Facebook sends several types of push notifications and is considering a new notification that tells a user when a friend plans to attend an event. You are asked to evaluate whether notifications are helping users or creating harm.
Constraints & Assumptions
Treat this as a product analytics and experimentation question, not a notification-ranking implementation problem.
Assume notification delivery, opens, dismissals, app sessions, event interactions, settings changes, and retention can be logged at user and notification-type level.
Preserve user experience: a notification can increase clicks while still being harmful if it causes opt-outs, churn, or poor long-term engagement.
Do not assume the new event notification is useful until demand and incremental value are measured.
Clarifying Questions to Ask Guidance
Is the evaluation for one existing notification type, all notification types, or the new event notification?
What is the main product objective: event discovery, meaningful interactions, app retention, or notification quality?
Are we allowed to run randomized experiments, or must we rely on observational data?
Which users are eligible for the event notification, and how frequently could they receive it?
Part 1 - Evaluate Existing Notifications
Which data and metrics would you use to determine whether an existing notification is helpful or harmful to users?
User-level outcomes such as sessions, retention, time to next session, notification setting changes, and long-term engagement.
Segmentation by notification type, user cohort, device, market, time of day, notification frequency, and relationship strength.
A distinction between short-term engagement and durable user value.
Part 2 - Decide Whether to Launch the Event Notification
What analyses or experiments would you run to decide whether to launch the friend-attending-event notification?
What This Part Should Cover Guidance
Pre-launch opportunity sizing and relevance checks using event attendance, friend graph, event browsing, and RSVP behavior.
A randomized experiment or holdout design that measures incremental event engagement rather than raw notification opens.
Guardrail metrics for notification fatigue, opt-outs, hides, uninstalls, negative feedback, and cannibalization of other surfaces.
Launch criteria that balance primary lift, statistical confidence, user harm, and heterogeneous treatment effects.
What a Strong Answer Covers Guidance
A strong answer builds a metric hierarchy, separates notification-level and user-level measurement, proposes a credible experiment, and explains how to interpret trade-offs when clicks rise but quality or retention falls.
Follow-up Questions Guidance
How would your plan change for new users versus highly active users?
How would you measure notification fatigue over several weeks?
If the experiment improves event RSVPs but increases notification opt-outs, what would you recommend?