Measure network effects and spillovers via experiments
Overview
This question evaluates experimental design, causal inference, and network analysis competencies—specifically the ability to define and identify direct, indirect, and total effects under interference using exposure mappings—falling under the Analytics & Experimentation domain and combining conceptual understanding of identification with practical application of clustered or graph‑randomized designs. It is commonly asked in technical interviews to probe reasoning about estimands, power and clustering, compliance and variance estimation, diagnostics for SUTVA violations, and operational and ethical constraints when implementing experiments on evolving social graphs.
Experiment design under network interference: direct and indirect effects
Context
You are evaluating a new social feature that can produce network spillovers (e.g., hashtag following, invite‑a‑friend prompts). Because one person's exposure can affect their neighbors, standard A/B tests that assume no interference are invalid. Design an experiment to identify and estimate direct effects on treated users and indirect/spillover effects on their neighbors.
Task
- Design
- Propose a two‑stage randomized design: (i) cluster‑level enablement, (ii) individual‑level encouragement (e.g., defaults or notifications). Alternatively, describe a graph cluster randomization.
- Define exposure conditions: None, Indirect‑only, Direct+Indirect. Explain how users move between exposure conditions over time as adoption spreads.
- Estimands
- Precisely define the Direct Effect, Indirect/Spillover Effect, and Total Effect using potential outcomes with exposure mappings.
- State the identifying assumptions and how you will check them.
- Power and clustering
- Explain how you will form graph clusters (algorithm, modularity target, size caps), manage overlap and cross‑cluster edges, and compute design effects for power.
- Analysis
- Specify intent‑to‑treat (ITT) vs treatment‑on‑the‑treated (TOT), how you’ll handle partial compliance, and variance estimators robust to clustering/network dependence.
- Describe diagnostics for SUTVA violations and saturation (dose‑response of neighbor exposure).
- Guardrails and ethics
- Define guardrails (abuse/integrity, feed health, notification fatigue) and fairness slices (locale, language, degree/centrality).
- State a decision rule if indirect effects are positive but direct effects are neutral.
- Operational challenges
- Address graph evolution mid‑experiment and asymmetric spillovers from large influencers.
- Propose instrumentation and a reweighting or re‑cluster strategy to keep estimates unbiased.
Solution
Login required