DoorDash plans to test a new order-batching/dispatch algorithm in 10 cities during August–September 2025. Spillovers between nearby areas are likely. Design and analyze the experiment: 1) Randomization unit: Propose a geo-cell clustering and stratified randomization plan that limits interference (e.g., 2–5 km hex cells), ensuring balance on baseline order volume, cuisine mix, and dasher supply. How will you detect and mitigate cross-cell spillovers? 2) Primary/secondary metrics: Choose one primary success metric (e.g., orders delivered per dasher hour, or 90th percentile delivery time) and at least three guardrails (e.g., customer cancellations, courier wait-at-store, restaurant prep SLA breaches, fairness across neighborhoods). Define each precisely, including inclusion/exclusion rules and winsorization. 3) Power: Baseline P90 delivery time is 40.0 minutes with SD 7.0; you expect a −1.2 minute improvement. There are 50 geo-cells per arm with average 8,000 orders per cell over the test. Intracluster correlation (ICC) is 0.20. Compute required sample size or achieved power using cluster-robust approximations; state any design effects and assumptions. 4) Analysis: Specify the intention-to-treat model with cluster-robust SEs, include pre-period CUPED covariates, and a plan for SRM checks. Provide the exact regression you would run (formula and covariates), how you will handle right-skew (e.g., log transform or quantile regression), and how you will aggregate cell-level quantiles. 5) Heterogeneity: Pre-register subgroup analyses (e.g., by time-of-day, cuisine, weather severity index). Show how you will control false discovery. What minimum subgroup sample size do you require? 6) Operational rollout: Describe a safe ramp plan and “kill switches.” If a mid-test storm hits 3 cities, explain how you’ll use difference-in-differences or synthetic control on impacted cells without biasing the ITT. 7) Decision rule: Write the exact thresholding rule (effect size, confidence, and guardrail constraints) for shipping, with an example calculation using hypothetical results.

This question evaluates experiment design and causal inference competencies—covering geo-randomization and spillover control, precise metric specification with guardrails, cluster-based power and sample-size calculations, intention-to-treat analysis with cluster-robust inference, heterogeneity analysis, and operational rollout and decision-rule planning in the Analytics & Experimentation domain. It is commonly asked because interviewers need to assess both conceptual understanding and practical application: designing robust cluster-randomized geo-experiments that limit interference, define and pre-register metrics and analysis, compute cluster-adjusted power, and specify operational safeguards and clear shipping criteria.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at DoorDash.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at DoorDash during technical interviews.

Design and analyze batching algorithm experiment

Experiment Design: New Order-Batching/Dispatch Algorithm (Aug–Sep 2025)

You are tasked with designing and analyzing a geo-experiment for a new order-batching/dispatch algorithm across 10 cities, where nearby areas may interfere with each other (e.g., dashers traveling across borders, demand shifting). The test runs for August–September 2025.

Provide a detailed plan for each item below.

Randomization unit and spillover control

Propose a geo-cell construction (e.g., 2–5 km hex cells) and a clustering/assignment scheme that limits interference across treatment and control.
Describe stratified randomization to balance baseline order volume, cuisine mix, and dasher supply.
Explain how you will detect spillovers across cells and how you’ll mitigate them in design and analysis.

Primary and secondary metrics

Choose exactly one primary success metric (e.g., Orders Delivered per Dasher Hour (ODDH) or P90 delivery time) and at least three guardrails (e.g., customer cancellations, courier wait-at-store, restaurant prep SLA breaches, fairness across neighborhoods).
Define each metric precisely, including:
- Inclusion/exclusion rules (what orders are in-scope, how to treat outliers, outages, bad telemetry).
- Any winsorization or trimming and at what cutoffs (e.g., 1st/99th percentiles).

Power and sample size

Baseline P90 delivery time = 40.0 minutes with SD = 7.0 minutes; expected improvement = −1.2 minutes.
There are 50 geo-cells per arm with an average of 8,000 orders per cell over the test; intracluster correlation (ICC) = 0.20.
Compute required sample size or achieved power using cluster-robust approximations. Show formulas, design effects, and assumptions.

Analysis plan (ITT)

Specify the intention-to-treat model with cluster-robust standard errors.
Include how you will use pre-period CUPED covariates.
Describe SRM checks.
Provide the exact regression you would run (formula and covariates), how you will handle right-skew (e.g., log transform or quantile regression), and how you will aggregate cell-level quantiles if needed.

Heterogeneity

Pre-register subgroup analyses (e.g., by time-of-day, cuisine, weather severity index).
Explain how you will control the false discovery rate and the minimum subgroup sample size required for valid inference.

Operational rollout and shocks

Describe a safe ramp plan and concrete kill switches.
If a mid-test storm hits 3 cities, explain how you’ll use difference-in-differences or synthetic control on impacted cells without biasing the ITT.

Decision rule

Write the exact shipping rule (effect size, statistical confidence, and guardrail constraints).
Include a worked example with hypothetical results.

Experiment Design: New Order-Batching/Dispatch Algorithm (Aug–Sep 2025)

Provide a detailed plan for each item below.

Randomization unit and spillover control

Propose a geo-cell construction (e.g., 2–5 km hex cells) and a clustering/assignment scheme that limits interference across treatment and control.
Describe stratified randomization to balance baseline order volume, cuisine mix, and dasher supply.
Explain how you will detect spillovers across cells and how you’ll mitigate them in design and analysis.

Primary and secondary metrics

Choose exactly one primary success metric (e.g., Orders Delivered per Dasher Hour (ODDH) or P90 delivery time) and at least three guardrails (e.g., customer cancellations, courier wait-at-store, restaurant prep SLA breaches, fairness across neighborhoods).
Define each metric precisely, including:
- Inclusion/exclusion rules (what orders are in-scope, how to treat outliers, outages, bad telemetry).
- Any winsorization or trimming and at what cutoffs (e.g., 1st/99th percentiles).

Power and sample size

Baseline P90 delivery time = 40.0 minutes with SD = 7.0 minutes; expected improvement = −1.2 minutes.
There are 50 geo-cells per arm with an average of 8,000 orders per cell over the test; intracluster correlation (ICC) = 0.20.
Compute required sample size or achieved power using cluster-robust approximations. Show formulas, design effects, and assumptions.

Analysis plan (ITT)

Specify the intention-to-treat model with cluster-robust standard errors.
Include how you will use pre-period CUPED covariates.
Describe SRM checks.
Provide the exact regression you would run (formula and covariates), how you will handle right-skew (e.g., log transform or quantile regression), and how you will aggregate cell-level quantiles if needed.

Heterogeneity

Pre-register subgroup analyses (e.g., by time-of-day, cuisine, weather severity index).
Explain how you will control the false discovery rate and the minimum subgroup sample size required for valid inference.

Operational rollout and shocks

Describe a safe ramp plan and concrete kill switches.
If a mid-test storm hits 3 cities, explain how you’ll use difference-in-differences or synthetic control on impacted cells without biasing the ITT.

Decision rule

Write the exact shipping rule (effect size, statistical confidence, and guardrail constraints).
Include a worked example with hypothetical results.

Design and analyze batching algorithm experiment

Quick Overview

Experiment Design: New Order-Batching/Dispatch Algorithm (Aug–Sep 2025)

Solution

Submit Your Answer to Earn 20XP

Design and analyze batching algorithm experiment

Quick Overview

Experiment Design: New Order-Batching/Dispatch Algorithm (Aug–Sep 2025)

Solution

Submit Your Answer to Earn 20XP