Top Dasher Program: Decision Framework, Experiment with Interference, Anti-Gaming, and Ethics

Context

You are a data scientist at a food delivery marketplace. Product proposes a "Top Dasher" program that grants the top 10% of drivers priority dispatch and flexible scheduling if they maintain an acceptance rate ≥ 70% and a completion rate ≥ 95% over the last 30 days.

Assume: the program is an overlay on the existing dispatch algorithm (priority bump in ranking), and flexible scheduling means earlier access to shift slots in constrained markets.

Tasks

1. Decision framework

• Should we launch? Build a framework that quantifies expected value and risks.
• Identify primary success metrics (e.g., order ETAs, fulfillment rate, cancellations, driver earnings dispersion, merchant and consumer NPS) and guardrails.
• Define an explicit go/no-go decision rule.

2. Experiment design with marketplace interference

• Design an experiment to estimate causal impact given that priority reallocates orders among drivers (violating standard SUTVA).
• Specify unit of randomization (driver, zone, market-day), clustering, sample size and duration.
• Explain how to mitigate SUTVA violations and spillovers.

3. Gaming and selection bias

• How will you prevent gaming (e.g., acceptance-rate manipulation) and selection bias?
• Propose instrumentation and a pre-registered analysis plan, including CUPED or pre-period covariate adjustment.

4. Heterogeneity

• Outline heterogeneity analyses (new vs. veteran drivers, peak vs. off-peak, high vs. low supply markets) and how their results would change the rollout plan.

5. Ethics and operations

• Discuss ethical and operational tradeoffs (fairness across drivers, small-market impacts, earnings volatility).
• Propose guardrail thresholds and a rollback plan.