Network Effects And Interference

What's being tested

Interviewers probe whether you can design and analyze randomized experiments when units are connected—so treatment can affect others (i.e., interference). They expect a Data Scientist to pick the correct randomization unit, define clear estimands (direct vs. spillover), compute power under clustering, and plan analysis that yields unbiased or interpretable causal contrasts. The goal is practical: produce an experiment and analysis plan a product and metrics team can execute and trust.

Core knowledge

• Interference / SUTVA: The Stable Unit Treatment Value Assumption fails when one unit’s outcome depends on others’ treatments; explicitly state where SUTVA is violated and how you’ll model it.

• Partial interference: Often assume interference only within clusters (e.g., city, listing neighborhood), enabling cluster randomization and tractable analysis; justify cluster boundaries using marketplace flows.

• Randomization unit tradeoff: Randomize at the individual, edge, or cluster level. Larger clusters reduce contamination but increase variance; increasing cluster count is usually better than increasing cluster size.

• Design effect and ICC: Design effect DE = 1 + (m − 1)·ρ, where m is average cluster size and ρ is intraclass correlation (ICC). Effective sample size ≈ N / DE; use this in sample-size calculations.

• Two-stage (hierarchical) randomization: Randomize clusters to different treated fractions, then randomize individuals within clusters—lets you estimate direct and spillover effects separately and improves power for both.

• Exposure mapping: Define an exposure function e_i = f(Z_neighbors, Z_i) that maps global assignment vector to the local exposure status used in estimand definitions (e.g., "treated & ≥1 treated neighbor").

• Estimands: Pre-specify direct effect (difference when unit treated vs control, holding neighbors’ assignments specified), indirect/spillover effect (effect on control units when neighbors are treated vs not), and total effect.

• Estimators: Use Horvitz–Thompson / inverse-probability weighting for arbitrary assignment schemes; for cluster randomization, difference-in-means across clusters or cluster-level regressions with cluster-robust standard errors are common.

• Randomization inference: Prefer exact or permutation tests for hypothesis testing under interference; preserves Type I control when asymptotics are unclear.

• Adjustment & covariates: Pre-stratify or block on network features (degree, community), and use covariate adjustment (ANCOVA) to reduce variance; retain randomization-based inference.

• Power calculations under interference: Simulate using realistic network, treatment assignment, and assumed spillover parameters; analytic formulas often insufficient when degree heterogeneity is large.

• Monitoring & integrity: Track assignment fidelity, cross-unit contamination, and time-varying exposure; pre-specify treatment exposure windows and handling of late entries/exits.

Worked example — Design and Analyze Airbnb Locker Experiment

First 30 seconds: clarify scope and metric — is the goal increased bookings, conversions, or host utilization? Ask which population is eligible (entire marketplace, select cities), expected adoption rate, and primary interference paths (guest-to-guest, host recommendations). Skeleton of answer: (1) define estimands: direct effect on treated listings, spillover effect on nearby listings, total marketplace effect; (2) choose randomization unit: cluster by geography or demand-supply neighborhood to contain guest flows; (3) sample-size and power: estimate ICC from historical conversion by neighborhood, compute DE and required number of clusters; (4) analysis plan: specify exposure mapping (e.g., treated listing, neighbor treated fraction), estimator (cluster-level differences + IPW for heterogeneous exposure), and randomization inference for p-values. Key tradeoff: choose cluster size — larger clusters reduce contamination but require more clusters for power; explicitly justify with DE calculations and simulation of spillover magnitude. Close by saying what you’d do with more time: run network simulations with real booking graphs to tune assignments and pre-register subgroup analyses and monitoring rules.

A second angle — Design a network-aware Wi‑Fi badge experiment

Here the network is explicit and dynamic: badges influence nearby peers in real time and hubs (conference speakers) create heavy-tailed degree distributions. Emphasize short treatment windows and time-varying exposure mapping (neighbors change over the session). Prefer randomized exposure at the edge or use graph cluster randomization where clusters are conference rooms or session groups. Use two-stage randomization to estimate peer contagion vs direct badge effect. Analysis must handle high degree heterogeneity (weight by degree, use IPW), and hypothesis testing should rely on permutation within clusters or network-aware randomization inference to control Type I error.

Common pitfalls

Pitfall: Ignoring interference and reporting a naïve ATE from individual randomization. This yields biased estimates because treated neighbors change counterfactuals; instead, explicitly define exposure values and estimands.

Pitfall: Choosing clusters by convenience (e.g., city boundaries) without checking actual marketplace flows. This can leave strong cross-cluster edges and uncontrolled contamination; use historical interaction/booking graphs to validate cluster boundaries or simulate.

Pitfall: Relying exclusively on asymptotic cluster-robust SEs with few clusters. With small cluster counts, use randomization inference or report permutation-based p-values and emphasize effect sizes and confidence intervals from IPW/Hajek estimators.

Connections

Interfering experiments often lead interviewers to pivot to heterogeneous treatment effects (how spillovers vary by degree or listing type), graph clustering / community detection methods used to form clusters, and sequential experimentation / adaptive designs when treatments and networks evolve over time.

Further reading

• Hudgens & Halloran (2008) — Toward Causal Inference With Interference — foundational framework for direct and indirect effects under interference.

• Ugander et al. (2013) — Graph Cluster Randomization: Designing Experiments on Networks — practical methods to form clusters and analyze networked A/B tests.