Design experiment for unconnected content in feed
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Part A — Validate that friends' content is more "social" than unconnected content in a personalized feed. Using impression- and interaction-level logs like feed_impressions(user_id, post_id, impression_ts) and interactions(user_id, post_id, type, interaction_ts), plus a friendships graph, design an analysis plan that: (i) defines a 'socialness' outcome at the impression level (e.g., a weighted action score per impression, follow/DM initiation within 7 days, or unique commenters); (ii) justifies and stress-tests action weights (like=1, comment=3, share=5) via sensitivity analysis and backtesting; (iii) controls for confounding from ranking/exposure (e.g., inverse propensity weighting or matched sampling using predicted exposure scores); (iv) specifies primary effect estimates (ATE and quantile effects overall and by cohort), statistical tests, CIs, and how you'll handle repeated measures per user.
Part B — We are launching unconnected content into a feed that previously showed only friends. Define success and design an A/B experiment that minimizes network interference: specify unit of randomization (viewer-level vs cluster), exposure definition, ramp plan, sample-size/power target (include MDE assumptions), primary KPIs (e.g., per-impression socialness, viewer retention D+1, sessions/user), and guardrails (cannibalization of friends' impressions/engagement, creator follows, ads RPM/CTR, integrity metrics). Describe how you'll detect and mitigate spillover (e.g., two-sided markets, creator supply responses), novelty/learning effects, and how you would interpret conflicting metric movements. Include decision thresholds and a rollback/ship framework.
Overview: This question evaluates a data scientist's competency in causal inference, observational analysis, metric design, and experiment engineering for personalization, specifically measuring and comparing the "socialness" of friend versus unconnected content.
Read the full Meta Data Scientist interview experience this question came from