Program Evaluation: Allow Car Dashers to Also Use Their Own Bikes/E-bikes
Context
DoorDash (DD) will relaunch an opt-in feature that lets existing car dashers also use their own bicycles/e-bikes for deliveries while retaining access to car mode. You must decide whether to scale this program next quarter. Design a rigorous experiment and decision framework that accounts for opt-in, operational constraints, and marketplace spillovers.
Task
Answer precisely and completely:
-
Business goal and hypotheses
-
State the primary business goal.
-
State one falsifiable primary hypothesis (example: reduce median delivery ETA in dense zones without reducing dasher earnings/hour).
-
List at least two secondary goals (e.g., cost per order, supply density).
-
Metrics
-
Define one primary success metric, at least three guardrail metrics, and two driver metrics.
-
For each metric, specify: exact definition (numerator/denominator where applicable), attribution window, unit of analysis, exclusions (e.g., batched orders, outliers), and directionality (increase/decrease desired).
-
Experimental design and randomization unit
-
Choose a randomization unit: dasher-level, zone–day-level, or order-level.
-
Justify your choice by addressing: noncompliance (opt-in, vehicle switching), spillovers/interference (supply rebalancing, congestion), inventory constraints (e.g., only 600 bikes for 1,200 interested dashers across 5 cities), and operational feasibility.
-
If opt-in is required, design an encouragement RCT (invite vs. no-invite). Explain ITT vs. TOT and how you’ll instrument actual bike usage.
-
Sample size and power
-
Outline your power plan: baseline assumptions, target MDE for the primary metric, clustering/ICC impact, seasonality/day-of-week controls, and how you’ll handle unequal cluster sizes. No detailed math needed, but be concrete about inputs you’d request.
-
Pre-rollout plan
-
Specify instrumentation to infer actual vehicle per trip, data QA checks, eligibility rules, safety/compliance, and a canary + ramp schedule with explicit stop/go thresholds.
-
Analysis plan
-
Describe how you’ll handle heterogeneous effects (e.g., hills/elevation, weather, time-of-day, restaurant wait), contamination/crossovers, and selection bias from opt-in.
-
Explain how to separate the speed-of-travel effect from restaurant latency.
-
Decision framework
-
Specify success criteria that trigger scale-up vs. rollback and how you’ll monitor post-launch with long-term holdouts.