Design Experiments to Evaluate Courier Initiatives Effectively
Overview
This question evaluates a candidate's proficiency in experimental design, causal inference, A/B testing, metric selection, statistical power analysis, and marketplace analytics for two-sided delivery platforms.
Scenario
You operate a two‑sided delivery marketplace with independent couriers (Dashers). The team must decide on three courier-related initiatives:
- Launch a Top Dasher program that routes higher‑quality orders (e.g., shorter distance, higher pay reliability) to top performers.
- Add extra pay incentives to raise Dasher engagement.
- Switch Dasher compensation from per‑order to per‑time.
Questions
1) Top Dasher program
How would you design an experiment to evaluate the Top Dasher program and determine whether it should be launched? Which metrics and success criteria would you track?
2) Response rate and extra pay incentives
a) How would you precisely define Dasher response rate?
b) What product/ops levers could improve it?
c) How would you structure and analyze an A/B test for the extra pay incentive program?
3) Time pay vs. order pay
Compare paying Dashers by time versus by completed orders: outline the pros and cons of each model and propose an empirical test to decide which is preferable.
Hint: Discuss metric selection, experiment design, statistical power, potential biases (e.g., marketplace spillovers), and business trade‑offs.
Solution
Login required