Design experiments under network interference
Overview
This question evaluates a data scientist's competence in experimental design and causal inference under network interference, assessing understanding of clustering strategies, exposure and exclusion definitions, contamination control, design effect and sample-size implications, estimation of intra-cluster correlation, and mixed-effects analysis for evolving clusters. Commonly asked in Analytics & Experimentation interviews to probe the ability to preserve internal validity while managing operational feasibility in two-sided marketplaces, it tests both conceptual understanding of interference and practical application skills in design and analysis planning.
A/B Test Design for Search-Ranking in a Two-Sided Marketplace with Interference
Context
You need to evaluate a change to the search-ranking algorithm in a two-sided marketplace where:
- Sellers can affect each other (cross-seller spillovers, e.g., inventory or ranking shifts that alter other sellers' exposure).
- Consumers can influence each other (social contagion, e.g., shares, word-of-mouth).
Design an interference-aware experiment that preserves internal validity while remaining operationally feasible.
Tasks
- Choose a randomization strategy: cluster randomization (e.g., by seller or geography) vs. ego-cluster randomization. Justify the choice given cross-seller spillovers and consumer social contagion.
- Precisely define exposure and exclusion criteria to control interference.
- Compute design effect and sample-size inflation for clustering with mean cluster size m = 50 and intra-cluster correlation ρ = 0.02.
- Explain how to estimate ρ pre-experiment from historical data.
- Describe how to prevent treatment contamination during reassignment and how to validate success using invariant metrics and sample-size ratio mismatch (SSRM) under clustering.
- If clusters evolve over time, propose a rehashing (re-clustering) schedule and an analysis plan using mixed-effects models with cluster random intercepts.
Solution
Login required