Design an experiment for order batching
Overview
This question evaluates competence in experiment design and causal inference for marketplace interventions, covering choice of randomization unit and mitigation of interference/spillovers, stratification and covariate adjustment, power and sample‑size calculations, detection of novelty/steady‑state effects, heterogeneity estimation, and pre-registered guardrail decision rules. It is commonly asked in the Analytics & Experimentation domain because interviewers need to assess both conceptual understanding of causal inference and the practical application of randomized designs and operational measurement to ensure robust, contamination‑aware evaluation of policy changes.
Experiment Design: Two-Order Batching Policy During Peak Hours
Context
DoorDash plans to test a dispatch policy that allows a dasher to pick up two nearby orders in one trip during peak hours. The goal is to estimate the causal impact on:
- (a) Customer order conversion rate
- (b) ETA accuracy (actual vs. quoted)
- (c) Dasher hourly earnings
- (d) Restaurant prep-time congestion
Assume batching is only enabled for eligible orders (e.g., distance and timing constraints) and that peak hours are predefined per market. You must account for marketplace spillovers (dashers, restaurants, and customers interact within local zones).
Precisely answer
- Choose a unit of randomization (e.g., zone-hour, store, dasher). Justify in light of interference/spillovers and explain how you will mitigate cross-treatment contamination.
- Specify a stratification and covariate-adjustment plan (e.g., city, cuisine, distance bands, forecasted demand) and how you will pre-register guardrail metrics (late deliveries, cancellations, CSAT).
- Provide sample size and duration: outline MDE, baseline rates, variance assumptions, and a power calculation at 80% power. Justify any sequential testing or group-sequential design for early stopping.
- Describe how to detect novelty/learning effects, discount the first-N days, and measure steady-state.
- Explain how to estimate heterogeneity (city-level and distance-band) without p-hacking (e.g., hierarchical modeling, shrinkage).
- Provide a precise promotion rule when guardrails worsen but primary metrics improve, including acceptable deltas and confidence thresholds.
Solution
Login required