Design a pricing experiment with network effects
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Scenario
You want to launch a **new pricing model** that incentivizes shoppers to place/pick up more orders during **rush hours** in a two-sided marketplace (supply and demand interact). You suspect **network effects / interference**: changing prices for some users may affect availability, ETAs, or acceptance rates for others.
## Task
Design an experiment to evaluate the new pricing model.
### Constraints
- A standard user-level A/B test may be invalid due to **spillovers** (interference) across users.
- The marketplace has heterogeneous geographies with different baselines.
### Requirements
Your design should include:
1. **Unit of randomization** and why (e.g., geo/market-level).
2. How you will choose **treatment/control markets** (e.g., matched pairs / lookalikes).
3. **Primary metric** (north star) and a set of diagnostic + guardrail metrics.
4. How you handle **bias/confounding** (seasonality, pre-trends, market differences).
5. Ramp plan, duration, and how you’ll estimate power/MDE at a high level.
6. Risks: spillovers across nearby markets, partial compliance, concurrent changes.
### Output
Provide a clear experimental plan and analysis approach (e.g., difference-in-differences).
Overview: This question evaluates experimental design and causal inference competencies for a two-sided marketplace pricing change, including handling network effects/interference, choice of unit of randomization and treatment markets, metric selection and diagnostics, power/MDE estimation, and operational risk assessment within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to assess practical application of randomized and quasi-experimental methods across heterogeneous geographies and operational constraints, requiring both hands-on experimental design skills and conceptual understanding of bias, spillovers, and diagnostic metrics.
Read the full Instacart Data Scientist interview experience this question came from