Design experiment for unconnected content in feed
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Part A — Validate that friends' content is more "social" than unconnected content in a personalized feed. Using impression- and interaction-level logs like feed_impressions(user_id, post_id, impression_ts) and interactions(user_id, post_id, type, interaction_ts), plus a friendships graph, design an analysis plan that: (i) defines a 'socialness' outcome at the impression level (e.g., a weighted action score per impression, follow/DM initiation within 7 days, or unique commenters); (ii) justifies and stress-tests action weights (like=1, comment=3, share=5) via sensitivity analysis and backtesting; (iii) controls for confounding from ranking/exposure (e.g., inverse propensity weighting or matched sampling using predicted exposure scores); (iv) specifies primary effect estimates (ATE and quantile effects overall and by cohort), statistical tests, CIs, and how you'll handle repeated measures per user.
Part B — We are launching unconnected content into a feed that previously showed only friends. Define success and design an A/B experiment that minimizes network interference: specify unit of randomization (viewer-level vs cluster), exposure definition, ramp plan, sample-size/power target (include MDE assumptions), primary KPIs (e.g., per-impression socialness, viewer retention D+1, sessions/user), and guardrails (cannibalization of friends' impressions/engagement, creator follows, ads RPM/CTR, integrity metrics). Describe how you'll detect and mitigate spillover (e.g., two-sided markets, creator supply responses), novelty/learning effects, and how you would interpret conflicting metric movements. Include decision thresholds and a rollback/ship framework.
Quick Answer: This question evaluates a data scientist's competency in causal inference, observational analysis, metric design, and experiment engineering for personalization, specifically measuring and comparing the "socialness" of friend versus unconnected content.