Experiment Design: Network-Aware Test for a "Friend Interaction Boost" in Feed Ranking

Context

You plan to ship a ranking change that boosts items in the feed when a user's friends have interacted with them. Because interactions propagate through the social graph, the Stable Unit Treatment Value Assumption (SUTVA) is violated under user-level randomization: a user's outcome can be affected by neighbors' assignments. Design a credible experiment that estimates causal lift under network interference.

Tasks

1. Randomization Unit

• Choose and defend one of: graph-cluster randomization (e.g., Louvain or partitioning), ego-network clustering, or geo/time switchbacks.
• Define how you will quantify and cap:
  • Cross-cluster edge cut ratio.
  • Exposure contamination across treatment arms.

2. Metrics

• Define primary metrics (e.g., session time, meaningful interactions) and guardrail metrics (e.g., spam reports, creator revenue cannibalization).
• Specify exposure-weighted metrics for partially treated users.

3. Power

• Given average cluster size m and intracluster correlation ρ, derive the effective sample size: n_eff ≈ (K · m) / [1 + (m − 1)ρ] where K is the number of clusters.
• Explain how this changes required test duration relative to a user-level A/B test.

4. Analysis

• Outline cluster-robust variance estimation; use CUPED with pre-period outcomes.
• Define intent-to-treat (ITT) vs exposure-on-treated estimands and how to estimate each.
• Describe how to handle creators whose audiences span both treatment and control clusters.

5. Diagnostics and Fallbacks

• Pre-commit spillover checks, negative controls, and a holdout of high-degree nodes.
• If contamination is too high mid-test, propose a redesign that preserves inference while limiting blast radius.