Cannibalization and Ecosystem Causal Diagnostics
Asked of: Data Scientist
Last updated
-
What it is Cannibalization is when a new feature, surface, or campaign shifts usage or revenue away from your own existing ones instead of creating net new value. Ecosystem causal diagnostics are experiment designs and checks that separate true incremental lift from shifts and spillovers across surfaces, users, or marketplace sides.
-
Why interviewers ask about it At large platforms, a “win” on one surface (e.g., Reels, Search Ads) can quietly reduce feed time, organic clicks, or creator supply elsewhere, hurting portfolio KPIs. Data Scientists are expected to design experiments that isolate budgets/inventory, detect interference, and report net impact to company-level metrics—not just local lifts.
-
Core ideas to know
- Net lift = gross lift minus displaced usage/revenue across owned surfaces and channels.
- SUTVA breaks under spillovers; use network-aware designs (cluster/graph randomization, geo or switchback tests). (web.stanford.edu)
- Define success on portfolio metrics; include guardrails for other surfaces, creator/seller supply, latency, and revenue mix.
- Isolate budgets/inventory to avoid cross-bucket cannibalization; pace ads fairly, separate supply pools.
- Use reassigned-control ad-lift (“ghost ads”) to avoid budget and audience overlap bias. (papers.ssrn.com)
- Map exposure/interference (who affects whom); consider power loss and variance reduction via regression/exposure logging. (arxiv.org)
- Check persistence: novelty/learning curves can mask short-run cannibalization or reveal long-run recovery.
-
A common pitfall Candidates present a user-level A/B that shows a local KPI uptick, but they didn’t isolate budgets or inventory, so treatment starved control or moved demand between variants. They ignore cross-surface guardrails, so a feature that boosts session starts cannibalizes feed time or higher-margin conversions. They also assume independence between users or sellers, missing interference that requires cluster or geo designs. When asked to diagnose, they lack a plan for portfolio metrics, exposure mapping, or falsification tests.
-
Further reading
- Eckles, Karrer, Ugander — Design and Analysis of Experiments in Networks (JCI, 2017). Practical foundations for experimentation under interference; motivates cluster/network designs. (web.stanford.edu)
- Johnson, Lewis, Nubbemeyer — Ghost Ads (JMR working paper/SSRN, 2017). Reassigned-control methodology for measuring true ad incrementality without budget cannibalization. (papers.ssrn.com)
- Lobel et al. — Reducing Interference Bias in Online Marketplace Experiments (Management Science, Airbnb meta-experiment). Concrete evidence and guidance for cluster randomization in two‑sided markets. (pubsonline.informs.org)