Evaluate Dasher Initiatives with A/B Testing and Metrics

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Evaluate Dasher Initiatives with A/B Testing and Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Evaluate Dasher Initiatives with A/B Testing and Metrics

Company: DoorDash

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

##### Scenario As a product/analytics lead at a food-delivery marketplace you must evaluate several Dasher-facing initiatives (Top-Dasher prioritization, Extra-Pay incentives, and switching pay model from per-order to per-time) before deciding whether to launch them. ##### Question How would you assess whether the Top-Dasher program should be launched? 2) For an "extra pay" incentive aimed at improving Dasher engagement: a) what primary success metric(s) would you track? b) design an A/B test (including treatment, control, experiment length, sample-size, and guardrail metrics). 3) The company is considering switching Dasher compensation from per-order to per-time. What are the key pros & cons of each model, and how would you experimentally validate which model is better for marketplace health? ##### Hints Discuss causal identification, experiment vs. quasi-experiment trade-offs, KPI definition (accept rate, fulfillment time, retention), supply-demand balance, cost impact, and possible negative externalities.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Evaluate Dasher Initiatives with A/B Testing and Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/DoorDash
DoorDash logo
DoorDash
Aug 4, 2025, 10:55 AM
hardData ScientistOnsiteAnalytics & Experimentation
18
0

Evaluate Dasher Initiatives with A/B Testing and Metrics

Scenario

You are the product/analytics lead for a food-delivery marketplace. You must evaluate several Dasher-facing initiatives before deciding whether to launch them:

  • Top-Dasher prioritization: preferentially prioritize “top” couriers in dispatch.
  • Extra-Pay incentives: targeted pay boosts to increase engagement in specific zones/times.
  • Switching pay model: from per-order to per-time compensation.

Assumptions (for clarity):

  • “Top-Dasher” = a courier who meets defined reliability/quality thresholds (e.g., high completion rate, on-time rate, low cancel rate) and would receive higher dispatch priority in treatment areas.
  • “Marketplace health” blends outcomes across consumer, merchant, and courier experiences alongside unit economics.

Questions

  1. How would you assess whether the Top-Dasher program should be launched?
  2. For an extra-pay incentive aimed at improving Dasher engagement: a) What primary success metric(s) would you track? b) Design an A/B test (treatment/control, randomization unit, experiment length, sample size, and guardrail metrics).
  3. The company is considering switching Dasher compensation from per-order to per-time. What are the key pros and cons of each model, and how would you experimentally validate which model is better for marketplace health?

Hints

Discuss causal identification (experiments vs. quasi-experiments), KPI definition (accept rate, fulfillment time, retention), supply–demand balance, cost impact, and potential negative externalities (gaming, spillovers, fairness).

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More DoorDash•More Data Scientist•DoorDash Data Scientist•DoorDash Analytics & Experimentation•Data Scientist Analytics & Experimentation