Program Evaluation: Allow Car Dashers to Also Use Their Own Bikes/E-bikes

Context

DoorDash (DD) will relaunch an opt-in feature that lets existing car dashers also use their own bicycles/e-bikes for deliveries while retaining access to car mode. You must decide whether to scale this program next quarter. Design a rigorous experiment and decision framework that accounts for opt-in, operational constraints, and marketplace spillovers.

Task

Answer precisely and completely:

1. Business goal and hypotheses

• State the primary business goal.
• State one falsifiable primary hypothesis (example: reduce median delivery ETA in dense zones without reducing dasher earnings/hour).
• List at least two secondary goals (e.g., cost per order, supply density).

2. Metrics

• Define one primary success metric, at least three guardrail metrics, and two driver metrics.
• For each metric, specify: exact definition (numerator/denominator where applicable), attribution window, unit of analysis, exclusions (e.g., batched orders, outliers), and directionality (increase/decrease desired).

3. Experimental design and randomization unit

• Choose a randomization unit: dasher-level, zone–day-level, or order-level.
• Justify your choice by addressing: noncompliance (opt-in, vehicle switching), spillovers/interference (supply rebalancing, congestion), inventory constraints (e.g., only 600 bikes for 1,200 interested dashers across 5 cities), and operational feasibility.
• If opt-in is required, design an encouragement RCT (invite vs. no-invite). Explain ITT vs. TOT and how you’ll instrument actual bike usage.

4. Sample size and power

• Outline your power plan: baseline assumptions, target MDE for the primary metric, clustering/ICC impact, seasonality/day-of-week controls, and how you’ll handle unequal cluster sizes. No detailed math needed, but be concrete about inputs you’d request.

5. Pre-rollout plan

• Specify instrumentation to infer actual vehicle per trip, data QA checks, eligibility rules, safety/compliance, and a canary + ramp schedule with explicit stop/go thresholds.

6. Analysis plan

• Describe how you’ll handle heterogeneous effects (e.g., hills/elevation, weather, time-of-day, restaurant wait), contamination/crossovers, and selection bias from opt-in.
• Explain how to separate the speed-of-travel effect from restaurant latency.

7. Decision framework

• Specify success criteria that trigger scale-up vs. rollback and how you’ll monitor post-launch with long-term holdouts.