Define Metrics and Account for Network and Novelty Effects
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Meta’s notification system triggers optional in-app surveys to measure user sentiment after notifications are sent.
##### Question
Define the primary goal metric, at least one driver metric, and at least one guard-rail metric for evaluating the notification survey feature. How would you explicitly account for potential network effects and novelty effects when interpreting these metrics?
##### Hints
Clarify business objective, user experience risks, and externalities; propose measurement technique for network & novelty effects.
Overview: Meta experimentation prompt on notification-triggered in-app surveys, covering goal, driver, and guardrail metrics, nonresponse bias, network effects, novelty effects, holdouts, and survey fatigue.
Meta's notification system triggers optional in-app surveys to measure user sentiment after notifications are sent.
Constraints & Assumptions
The survey feature should improve measurement quality without harming user experience.
Users may ignore surveys, creating nonresponse bias.
Survey prompts may create network effects, model feedback loops, or short-term novelty effects.
Define at least one primary goal metric, one driver metric, and one guardrail metric.
Clarifying Questions to Ask Guidance
What decision will the survey signal inform: notification ranking, frequency capping, policy, or product quality?
Are surveys randomly sampled from notifications or targeted to certain users/events?
What response options are available, and is there free text?
Can we run a persistent holdout or cluster-level experiment?
What a Strong Answer Covers Guidance
A primary metric tied to measurement value, such as bias-adjusted sentiment coverage, usable sentiment responses per notification type, or precision of negative-sentiment estimates.
Driver metrics such as response rate, completion rate, coverage by notification type, time-to-response, sample representativeness, and effective sample size.
Guardrails such as notification opt-outs, survey dismissals, app exits, session length, retention, notification engagement, complaint rate, and survey fatigue.
Explicit handling of nonresponse and selection bias using random sampling, weighting, post-stratification, or holdouts.
Network effects: cluster randomization, social graph spillover checks, or monitoring whether survey-driven ranking changes affect connected users.
Novelty effects: persistent holdout, pre/post trend monitoring, ramp analysis, and separating launch-week spikes from steady-state behavior.
Interpretation plan that avoids overreacting to biased or low-precision sentiment.
Follow-up Questions Guidance
How would you estimate sentiment if only unhappy users respond?
What guardrail would stop the feature?
How would survey results feed back into notification ranking safely?
How would you distinguish novelty from a real product improvement?