Analytics & Experimentation Case: Socialness of Friends vs Unconnected Content
Context
You work on a personalized feed that shows posts from friends and unconnected creators. You have impression- and interaction-level logs:
-
feed_impressions(user_id, post_id, impression_ts, slot, session_id, device, locale, ranker_score, content_type, creator_id)
-
interactions(user_id, post_id, type, interaction_ts)
-
friendships(viewer_id, friend_id, edge_ts) indicating friendship ties at impression time
Assume you can join impressions to interactions and friendship status at the impression timestamp.
Part A — Observational Analysis Plan
Validate that content from friends is more "social" than content from unconnected creators in a personalized feed. Using the logs and the friendship graph, design an analysis that:
-
Defines a per-impression "socialness" outcome. Examples include:
-
A weighted action score within a post-impression window.
-
Follow or DM initiation within 7 days.
-
Unique commenters on the post attributable to the viewer.
-
Justifies and stress-tests action weights (e.g., like=1, comment=3, share=5) via sensitivity analysis and backtesting against long-term outcomes (e.g., retention, future sessions).
-
Controls for confounding from ranking and exposure differences between friend and unconnected impressions (e.g., inverse propensity weighting using predicted exposure scores, matched sampling on slot/score/context).
-
Specifies primary effect estimates (overall ATE and quantile effects), cohort cuts, statistical tests, confidence intervals, and how you will handle repeated measures per user and creator.
Part B — Experiment Design for Introducing Unconnected Content
You will launch unconnected content into a feed that previously showed only friends. Define success and design an A/B test that minimizes network interference. Specify:
-
Unit of randomization (viewer-level vs. graph/geo clusters) and the exposure definition.
-
Ramp plan and sample-size/power targets with explicit MDE assumptions.
-
Primary KPIs (e.g., per-impression socialness, D+1 retention, sessions/user) and guardrails (e.g., cannibalization of friends' impressions/engagement, creator follows, ads RPM/CTR, integrity metrics).
-
How you will detect and mitigate spillover/interference (two-sided creator–viewer market, creator supply responses), novelty/learning effects, and how to interpret conflicting metric movements.
-
Decision thresholds and a rollback/ship framework.