Measure a friend-recommendation launch
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
A new friend-recommendation algorithm ships behind a feature flag. Design how you will measure success and decide whether to launch:
- State no more than 3 primary metrics and 3 counter/guardrail metrics, with precise 14-day attribution windows (e.g., confirmed-friends-per-DAU, acceptance rate, qualified exposures-per-DAU; guardrails: spam reports, hide-rate, session length).
- Define the experimental unit and randomization layers (user-level, network clusters for interference) and how you’ll handle network effects (ghost/long-term holdouts or staggered rollouts).
- Suppose in the first 14 days: MAU +5% (p<0.05), engagement per user −3% (p<0.05), confirmed friends per DAU +2% (p=0.07). Make a launch/no-launch recommendation and justify with a quantitative decision rule.
- Translate a +2% lift in confirmed-friends-per-DAU into an estimated annual revenue/profit delta given assumed ARPU and margin; specify the minimum economically-meaningful effect that warrants a full rollout.
- Describe how you will monitor post-launch for regression and set automatic rollback criteria.
Overview: This question evaluates experiment design, metric selection, causal inference and business-metric translation for social features, including defining primary and guardrail metrics, choosing experimental units to mitigate network interference, and estimating revenue impact from observed lifts.