Network Effects And Cluster Randomization

What's being tested

These prompts evaluate a candidate's ability to design and analyze experiments when interference (units affecting each other) breaks the usual A/B test assumptions. Interviewers probe whether you can define a clear estimand, construct an exposure model, pick a randomization strategy (unit vs. cluster randomization), and quantify power and bias under spillovers. Meta cares because social products have strong network effects; a Data Scientist must measure causal impact while controlling contamination, false precision, and rollout risk.

Core knowledge

• Interference / spillover: occurs when one unit’s treatment affects another’s outcome; this violates SUTVA and requires explicit exposure mapping (e.g., treated fraction in neighborhood) to define causal effects unambiguously.

• Estimand types: average treatment effect (ATE) is insufficient; define direct, indirect (spillover), and total effects. Use potential-outcomes with exposure mapping: $Y_i(z_i, z_{N(i)})$.

• Exposure models: operationalize interference using k-hop treated proportion, nearest-neighbor treated indicator, or graph-cut exposures; choose parsimonious models to avoid combinatorial explosion of potential exposures.

• Cluster randomization: randomize connected subgraphs to treatment or control to reduce cross-arm edges; cluster size (m) and number (K) trade power vs. contamination risk; prefer many small clusters when ICC is low.

• Design effect and ICC: adjust variance by design effect $DE = 1 + (m-1)\,ICC$. Effective sample size ≈ $N/DE$. Estimate ICC from historical metrics or pilot.

• Graph clustering algorithms: use Louvain, Infomap, or spectral clustering to produce clusters; balance edge-cut minimization with cluster size constraints to avoid very large clusters that reduce power.

• Randomization schemes: cluster-level (reduces contamination), stratified (by community size or pre-period metric), and graph-based rerandomization (reject assignments with many cross-arm edges) are common; log computational cost for large graphs.

• Analysis methods: use cluster-robust SEs, permutation / randomization inference on clusters, and hierarchical models to separate within-cluster and between-cluster variability; for exposure models, estimate via regression with exposure covariates and robust SEs.

• Power & sample size: simulate using realistic network and behavior models; analytic approximations use $Var(\bar{Y}) \approx \sigma^2/(n\cdot p(1-p))\cdot DE$ for difference-in-means, adjusting for binary/zero-inflated outcomes and expected effect heterogeneity.

• Contamination and routing: measure cross-arm interactions post-hoc with edge-level logs (message edges, payment flows); quantify contamination rate and bias direction to inform rollout decisions and guardrails.

• Inference under interference: consider partial interference assumptions (clusters independent) or use randomization-based inference (permutation within cluster-assignments) when asymptotics are fragile.

• Safety & guardrail metrics: for monetization features (ARPU, fraud rate) include short windows for safety, pre-specified stopping rules, and correction for multiple looks (e.g., alpha-spending) when sequential monitoring is needed.

Worked example — "Design cluster-randomized test under network effects"

First 30 seconds: clarify the estimand (direct effect on treated users? total effect including spillovers?), define allowable exposure (e.g., treated user with ≥1 treated friend), and constraints (graph size, rollout risk). Skeleton answer pillars: (1) construct clusters by minimizing cross-arm edges using Louvain with size caps; (2) randomize clusters stratified by pre-period metric and cluster size; (3) specify estimands and exposure mappings and plan analysis (cluster-level difference-in-means + randomization inference); (4) run power sims under varying ICC and contamination; (5) define guardrails and monitoring. Key tradeoff: larger clusters reduce contamination but increase DE and reduce power — explicitly quantify via $DE=1+(m-1)ICC$ and show how many clusters needed for target power. Close by saying: if more time, I'd run synthetic-network simulations calibrated to historical interaction rates, test alternative exposure mappings, and prepare post-hoc diagnostics for contamination and heterogeneous effects.

A second angle — "Design and evaluate P2P payments in messaging"

The same interference concerns appear but monetization and fraud elevate priorities. Here the estimand often includes ARPU change and risk metrics (fraud rate, chargebacks). Network pathways (who pays whom) are directional, so exposure models must account for sender vs receiver roles. Cluster construction should preserve natural payment subgraphs (frequent payer-recipient pairs) and avoid splitting payment dyads across arms. Power sims must incorporate heavy-tailed payment amounts and low event rates; consider zero-inflated outcome models. Analysis should add outcome transformations (log or winsorization) and pre-specified subgroups (high-value transactors). Operationally, include conservative safety stops and offline fraud scoring as guardrails.

Common pitfalls

Pitfall: Over-simplifying interference to "ignore it" and running a unit-randomized A/B test — this yields biased estimates when cross-arm edges are non-negligible; always quantify expected contamination.

Pitfall: Treating cluster construction as a black box — using Louvain without size constraints can create a few giant clusters that kill power via high DE; present alternative clusterings and show sensitivity.

Pitfall: Reporting naïve p-values without accounting for cluster-level randomization or exposure mapping — use cluster-robust SEs or randomization inference and explicitly state the inferential assumptions.

Connections

Interviewers may pivot to causal graph identification, heterogeneous treatment effects (HTE) across network positions, or to implementation metrics like logging edge-level exposures in Postgres or event streams. They might also ask about simulation pipelines to calibrate power for graph experiments.

Further reading

• Ugander et al., "Graph Cluster Randomization" — practical cluster construction ideas for experiments on networks.

• Hudgens & Halloran, "Interference and Causal Inference" — foundational theory for partial interference and estimands.