Define Metrics and Account for Network and Novelty Effects
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Meta’s notification system triggers optional in-app surveys to measure user sentiment after notifications are sent.
##### Question
Define the primary goal metric, at least one driver metric, and at least one guard-rail metric for evaluating the notification survey feature. How would you explicitly account for potential network effects and novelty effects when interpreting these metrics?
##### Hints
Clarify business objective, user experience risks, and externalities; propose measurement technique for network & novelty effects.
Quick Answer: This question evaluates a data scientist's ability to define a primary goal metric plus driver and guard-rail metrics for an in-app feedback measurement system while explicitly accounting for network interference and temporal novelty effects on outcomes.