Design a pricing experiment with network effects
Quick Overview
This question evaluates experimental design and causal inference competencies for a two-sided marketplace pricing change, including handling network effects/interference, choice of unit of randomization and treatment markets, metric selection and diagnostics, power/MDE estimation, and operational risk assessment within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to assess practical application of randomized and quasi-experimental methods across heterogeneous geographies and operational constraints, requiring both hands-on experimental design skills and conceptual understanding of bias, spillovers, and diagnostic metrics.
Scenario
You want to launch a new pricing model that incentivizes shoppers to place/pick up more orders during rush hours in a two-sided marketplace (supply and demand interact). You suspect network effects / interference: changing prices for some users may affect availability, ETAs, or acceptance rates for others.
Task
Design an experiment to evaluate the new pricing model.
Constraints
- A standard user-level A/B test may be invalid due to spillovers (interference) across users.
- The marketplace has heterogeneous geographies with different baselines.
Requirements
Your design should include:
- Unit of randomization and why (e.g., geo/market-level).
- How you will choose treatment/control markets (e.g., matched pairs / lookalikes).
- Primary metric (north star) and a set of diagnostic + guardrail metrics.
- How you handle bias/confounding (seasonality, pre-trends, market differences).
- Ramp plan, duration, and how you’ll estimate power/MDE at a high level.
- Risks: spillovers across nearby markets, partial compliance, concurrent changes.
Output
Provide a clear experimental plan and analysis approach (e.g., difference-in-differences).