Track Success and Guardrail Metrics for Push Notifications
Quick Overview
Evaluates success and guardrail metrics for a push-notification feature in a networked travel app. Strong answers measure incremental itinerary engagement and retention, monitor opt-outs and fatigue, and design a cluster or graph-aware experiment to reduce spillovers.
Track Success and Guardrail Metrics for Push Notifications
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
##### Scenario
Designing and evaluating push notifications for a travel-recommendation mobile app (TripAdvisor-like).
##### Question
What primary success metrics would you track for a new push-notification feature and which guardrail metrics would you include (e.g., uninstalls, unsubscribes)? The app has strong network effects (users share itineraries). How would you design an A/B test for the notification while mitigating interference? Describe the unit of randomization and why.
##### Hints
Think engagement, retention, negative‐impact guardrails, and cluster-based randomization to reduce spillover.
Quick Answer: Evaluates success and guardrail metrics for a push-notification feature in a networked travel app. Strong answers measure incremental itinerary engagement and retention, monitor opt-outs and fatigue, and design a cluster or graph-aware experiment to reduce spillovers.
Track Success and Guardrail Metrics for Push Notifications
You are designing and evaluating a new push-notification feature for a travel-recommendation mobile app where users can create and share itineraries. Because users influence each other through sharing and engagement, the product may have network effects.
Constraints & Assumptions
Treat this as a product metrics and experiment-design question.
Assume logs exist for notification eligibility, send, delivery, open, click, itinerary actions, sharing, sessions, retention, unsubscribes, and uninstalls.
The notification should create incremental user value without causing notification fatigue or harming the broader network.
Experiment design should account for spillovers between connected users.
Clarifying Questions to Ask Guidance
What notification is being tested: itinerary reminder, recommendation, share prompt, or reactivation message?
Who is eligible, and how often can users receive it?
What is the main objective: itinerary creation, sharing, bookings, retention, or engagement?
How strong are social or network interactions in the product?
Part 1 - Define Success Metrics
What primary success metrics would you track for the new push-notification feature?
Fatigue metrics such as notification frequency, repeat exposures, declining open rate, and long-term retention.
Cannibalization of organic sessions or other channels.
Part 3 - Design a Network-Aware Experiment
Given the app's network effects, how would you design an A/B test to mitigate interference or spillovers?
What This Part Should Cover Guidance
Why individual-level randomization may create spillovers when users share itineraries or affect friends.
Cluster, geo, household, or graph-based randomization options and trade-offs.
Exposure logging, cluster balance, sample size, analysis of direct and network effects, and guardrail thresholds.
What a Strong Answer Covers Guidance
A strong answer defines success as incremental durable travel-planning value, monitors fatigue and harm, and proposes a randomization strategy that reduces spillovers while preserving enough power to make a launch decision.
Follow-up Questions Guidance
How would you choose clusters for randomization?
What if notification opens rise but retention falls?