Design Experiments to Evaluate Courier Initiatives Effectively

Quick Overview

Evaluates experiment design for courier marketplace initiatives with interference and network effects. Strong answers cover Top Dasher routing, pay incentives, per-time compensation, stakeholder metrics, guardrails, and rollout decisions.

Design Experiments to Evaluate Courier Initiatives Effectively

Company: DoorDash

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

##### Scenario The team must decide on three courier-related initiatives: ( 1) launch a Top Dasher program that routes higher-quality orders to top performers, ( 2) add extra pay incentives to raise Dasher engagement, and ( 3) switch Dasher compensation from per-order to per-time. ##### Question How would you design an experiment to evaluate the Top Dasher program and determine whether it should be launched? Which metrics and success criteria would you track? 2) How would you precisely define Dasher response rate, what levers could improve it, and how would you structure and analyze an A/B test for the extra pay incentive program? 3) Compare paying Dashers by time versus by completed orders: outline the pros and cons of each model and propose an empirical test to decide which is preferable. ##### Hints Discuss metric selection, experiment design, statistical power, potential biases, and business trade-offs.

Quick Answer: Evaluates experiment design for courier marketplace initiatives with interference and network effects. Strong answers cover Top Dasher routing, pay incentives, per-time compensation, stakeholder metrics, guardrails, and rollout decisions.

|Home/Analytics & Experimentation/DoorDash
DoorDash logo
DoorDash
Jul 12, 2025, 6:59 PM
hardData ScientistOnsiteAnalytics & Experimentation
90
0

Experiments for Courier Marketplace Initiatives

You operate a two-sided delivery marketplace with independent couriers. The team must evaluate three courier-related initiatives:

  1. Launch a Top Dasher program that routes higher-quality orders to top performers.
  2. Add extra pay incentives to raise courier engagement.
  3. Switch courier compensation from per-order pay to per-time pay.

Design experiments and metrics to evaluate these initiatives while accounting for marketplace spillovers.

Constraints & Assumptions

  • Treat the marketplace as multi-sided: couriers, customers, merchants, and platform economics can all be affected.
  • Interference is likely because order allocation and courier supply are shared within markets.
  • Define eligibility and treatment assignment before measuring outcomes.
  • Include safety, quality, and fairness guardrails.

Clarifying Questions to Ask Guidance

  • How is "top performer" defined, and is eligibility fixed before the experiment?
  • Which markets, courier types, order types, and time windows are in scope?
  • What are the available incentive budgets and operational constraints?
  • Are we optimizing courier engagement, customer experience, marketplace efficiency, or profit?

Part 1 - Top Dasher Program

Design an experiment to evaluate routing higher-quality orders to top performers and decide whether to launch.

What This Part Should Cover Guidance

  • Address selection bias by freezing eligibility before assignment.
  • Discuss cluster, market, zone, or switchback designs to handle interference.
  • Track top-courier outcomes, non-top-courier outcomes, customer outcomes, merchant outcomes, and platform economics.
  • Define launch criteria and harm guardrails for marketplace fairness and quality.

Part 2 - Response Rate and Extra Pay Incentives

Define courier response rate, identify levers to improve it, and design an A/B test for extra pay incentives.

What This Part Should Cover Guidance

  • Define numerator, denominator, offer window, eligible offers, and response deadline.
  • Consider levers such as pay transparency, incentive amount, batching, distance prediction, notification timing, and courier targeting.
  • Randomize at an appropriate unit and avoid contamination across couriers in the same market-time cell.
  • Measure response rate, acceptance, completion, lateness, cancellation, earnings, cost per incremental order, and retention.

Part 3 - Per-Time Pay Versus Per-Order Pay

Evaluate a compensation change from per-order to per-time pay.

What This Part Should Cover Guidance

  • Identify behavioral risks such as slower completion incentives, cherry-picking, or gaming.
  • Track courier earnings, active time, utilization, delivery time, cancellation, customer ETA, merchant wait, and contribution margin.
  • Use market-level or switchback designs if marketplace equilibrium changes.
  • Include longer-term retention, supply availability, and cost sustainability.

What a Strong Answer Covers Guidance

  • Recognizes marketplace interference and avoids naive user-level experimentation.
  • Defines clear metrics for every stakeholder group.
  • Separates short-term operational lift from long-term marketplace health.
  • Uses decision criteria that balance engagement, quality, fairness, and economics.

Follow-up Questions Guidance

  • How would you detect harm to non-top couriers?
  • What would you do if extra pay improves response rate but worsens unit economics?
  • How would you analyze spillovers between treated and untreated couriers?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More DoorDash•More Data Scientist•DoorDash Data Scientist•DoorDash Analytics & Experimentation•Data Scientist Analytics & Experimentation