Decide event notification launch via experiments
Overview
This question evaluates a data scientist's competency in experimentation design, causal inference under network interference, metric engineering, and cluster-aware power and ramp planning within the Analytics & Experimentation domain, with a level of abstraction that is predominantly practical application supported by conceptual understanding of randomization and interference. It is commonly asked to assess the ability to design randomized strategies that account for spillovers, specify primary outcomes and guardrails with precise attribution windows, handle interference and long-term holdouts, and produce pre-registered stopping rules and launch/no-launch criteria that balance overall lift against harms in sensitive cohorts.
Meta plans a new notification that tells you when friends are going to an event. Determine whether to launch it.
- Design the experiment accounting for network spillovers (friends influence friends): propose and justify a randomization strategy (e.g., ego-network clustering, geo holdouts, or friend-exposure exclusion) and how you'll measure direct, indirect, and total effects.
- Specify primary outcomes and guardrails with exact windows: event page visits (1d/7d), RSVPs/attendance, session depth, hides/mutes/opt-outs, notification fatigue (per-user notification count), and retention. Provide formulas and attribution rules when multiple notifications coincide.
- Define a ramp plan, sample-size/power assumptions (including cluster design effects), stopping boundaries, and success criteria that balance overall lift vs harms in sensitive cohorts (e.g., users who previously muted event notifications).
- Address interference with existing notification types, cold-start targeting, and long-term effects: propose long-term holdouts or sequential experiments, and how you'll check for novelty decay.
- Provide a clear launch/no-launch decision framework, including pre-registered thresholds and remediation steps if guardrails regress.