Analytics & Experimentation Case: Socialness of Friends vs Unconnected Content

Context

You work on a personalized feed that shows posts from friends and unconnected creators. You have impression- and interaction-level logs:

• feed_impressions(user_id, post_id, impression_ts, slot, session_id, device, locale, ranker_score, content_type, creator_id)
• interactions(user_id, post_id, type, interaction_ts)
• friendships(viewer_id, friend_id, edge_ts) indicating friendship ties at impression time

Assume you can join impressions to interactions and friendship status at the impression timestamp.

Part A — Observational Analysis Plan

Validate that content from friends is more "social" than content from unconnected creators in a personalized feed. Using the logs and the friendship graph, design an analysis that:

1. Defines a per-impression "socialness" outcome. Examples include:
   • A weighted action score within a post-impression window.
   • Follow or DM initiation within 7 days.
   • Unique commenters on the post attributable to the viewer.
2. Justifies and stress-tests action weights (e.g., like=1, comment=3, share=5) via sensitivity analysis and backtesting against long-term outcomes (e.g., retention, future sessions).
3. Controls for confounding from ranking and exposure differences between friend and unconnected impressions (e.g., inverse propensity weighting using predicted exposure scores, matched sampling on slot/score/context).
4. Specifies primary effect estimates (overall ATE and quantile effects), cohort cuts, statistical tests, confidence intervals, and how you will handle repeated measures per user and creator.

Part B — Experiment Design for Introducing Unconnected Content

You will launch unconnected content into a feed that previously showed only friends. Define success and design an A/B test that minimizes network interference. Specify:

1. Unit of randomization (viewer-level vs. graph/geo clusters) and the exposure definition.
2. Ramp plan and sample-size/power targets with explicit MDE assumptions.
3. Primary KPIs (e.g., per-impression socialness, D+1 retention, sessions/user) and guardrails (e.g., cannibalization of friends' impressions/engagement, creator follows, ads RPM/CTR, integrity metrics).
4. How you will detect and mitigate spillover/interference (two-sided creator–viewer market, creator supply responses), novelty/learning effects, and how to interpret conflicting metric movements.
5. Decision thresholds and a rollback/ship framework.