Geo and Clustered Experiments
Asked of: Data Scientist
Last updated
-
What it is Geo experiments randomize entire geographic units (e.g., Nielsen DMAs, cities, countries) into test vs. control to estimate incremental impact when user-level randomization is impractical. Clustered experiments randomize groups of interacting units (companies, stores, friend-graph clusters) to reduce spillovers that would bias standard A/B tests.
-
Why interviewers ask about it At companies like Meta and Google, ads and social features create strong interference across users and regions; you’re expected to know designs beyond vanilla user-randomized tests. They want to hear how you’d get unbiased estimates under privacy and platform constraints, trade off power vs. bias, and use tools like matched markets, synthetic control, or ego-clusters.
-
Core ideas to know
- Use cluster/geo randomization when SUTVA fails due to spillovers, auctions, or network effects.
- Improve balance with matched pairs; Google’s Trimmed Match and TBR are common for geos.
- Leverage pre-period covariates; CUPED or synthetic controls (e.g., GeoLift) tighten variance.
- Analyze at the cluster level; the sample size is the number of clusters, not users.
- Plan power with intracluster correlation; design effect inflates required sample size/duration.
- Build minimally “leaky” geos; cluster by mobility/traffic (e.g., GeoCUTS) and separate media buys.
- Check parallel trends and counterfactual fit; add guardrails for seasonality and shocks.
-
A common pitfall Candidates default to user-level A/B tests for ads, pricing, or feed ranking where users influence each other, yielding biased lift. They then compute power on millions of users instead of a few dozen clusters, massively underpowering the design. Analysis mistakes follow: mixing individual-level outcomes with cluster assignment and ignoring pre-period imbalance. Strong answers mention clustering/matching, cluster-level inference, interference diagnostics, and operational safeguards against geographic contamination and holiday effects.
-
Further reading
- Trimmed Match Design for Randomized Paired Geo Experiments (Google Research) — practical matched-markets design and analysis for geo tests; discusses small-N, heavy tails, and time variation. https://research.google/pubs/trimmed-match-design-for-randomized-paired-geo-experiments/
- GeoLift Methodology (Meta open source) — synthetic control–based geo experimentation with market selection and power calculators; useful when user-level randomization isn’t feasible. https://facebookincubator.github.io/GeoLift/docs/Methodology/
- Using Ego-Clusters to Measure Network Effects at LinkedIn (arXiv) — cluster designs on social graphs to handle interference; concrete, large-scale industry example. https://arxiv.org/abs/1903.08755
Related concepts
- Clustered And Networked Experiments
- Statistical Inference For Experiments
- Cluster Randomized Experiments And Network InterferenceAnalytics & Experimentation
- Network Interference And Cluster RandomizationAnalytics & Experimentation
- Causal Inference And Quasi-Experiments
- A/B Testing And Experiment Analysis