Measure a friend-recommendation launch

Quick Overview

This question evaluates experiment design, metric selection, causal inference and business-metric translation for social features, including defining primary and guardrail metrics, choosing experimental units to mitigate network interference, and estimating revenue impact from observed lifts.

Measure a friend-recommendation launch

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

A new friend-recommendation algorithm ships behind a feature flag. Design how you will measure success and decide whether to launch: - State no more than 3 primary metrics and 3 counter/guardrail metrics, with precise 14-day attribution windows (e.g., confirmed-friends-per-DAU, acceptance rate, qualified exposures-per-DAU; guardrails: spam reports, hide-rate, session length). - Define the experimental unit and randomization layers (user-level, network clusters for interference) and how you’ll handle network effects (ghost/long-term holdouts or staggered rollouts). - Suppose in the first 14 days: MAU +5% (p<0.05), engagement per user −3% (p<0.05), confirmed friends per DAU +2% (p=0.07). Make a launch/no-launch recommendation and justify with a quantitative decision rule. - Translate a +2% lift in confirmed-friends-per-DAU into an estimated annual revenue/profit delta given assumed ARPU and margin; specify the minimum economically-meaningful effect that warrants a full rollout. - Describe how you will monitor post-launch for regression and set automatic rollback criteria.

Overview: This question evaluates experiment design, metric selection, causal inference and business-metric translation for social features, including defining primary and guardrail metrics, choosing experimental units to mitigate network interference, and estimating revenue impact from observed lifts.

Meta logo
Meta
Oct 13, 2025
mediumData ScientistOnsiteAnalytics & Experimentation
5
0

A new friend-recommendation algorithm ships behind a feature flag. Design how you will measure success and decide whether to launch:

  • State no more than 3 primary metrics and 3 counter/guardrail metrics, with precise 14-day attribution windows (e.g., confirmed-friends-per-DAU, acceptance rate, qualified exposures-per-DAU; guardrails: spam reports, hide-rate, session length).
  • Define the experimental unit and randomization layers (user-level, network clusters for interference) and how you’ll handle network effects (ghost/long-term holdouts or staggered rollouts).
  • Suppose in the first 14 days: MAU +5% (p<0.05), engagement per user −3% (p<0.05), confirmed friends per DAU +2% (p=0.07). Make a launch/no-launch recommendation and justify with a quantitative decision rule.
  • Translate a +2% lift in confirmed-friends-per-DAU into an estimated annual revenue/profit delta given assumed ARPU and margin; specify the minimum economically-meaningful effect that warrants a full rollout.
  • Describe how you will monitor post-launch for regression and set automatic rollback criteria.