Diagnose LA completed-order drop and design experiment

Quick Overview

This question evaluates a data scientist's competency in funnel-based diagnostics, causal inference, and experimental design for distinguishing supply-side versus demand-side drivers in a two-sided marketplace, emphasizing metric selection, segmentation, and guardrail monitoring.

Diagnose LA completed-order drop and design experiment

Company: DoorDash

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

On the week of 2025-08-25, completed orders in Los Angeles dropped 12% versus the week of 2025-08-18. Early indicators: sessions +4% WoW, order creations +1%, accept rate fell from 94% to 88%, p95 ETA rose 42→58 minutes, and cancellations with reason = 'no_courier' increased from 1.5% to 5.8%. Restaurant availability, pricing, and marketing spend were unchanged. Most of the decline occurred 18:00–21:00 local time. Design a rigorous investigation and test plan that cleanly separates demand-side from supply-side causes and validates the true driver: A) Diagnosis: Lay out a funnel from session → order_created → accepted → picked_up → completed. Specify the exact cuts (by hour, ZIP, courier density, restaurant category, distance bands, surge level), sanity checks for tracking/config regressions, and the minimal set of plots/tables you would produce to prove whether supply is the bottleneck. What additional metrics would you compute, and what thresholds would convince you? B) Causal validation: If supply shortage during the dinner peak is your leading hypothesis, propose an experiment to fix it (e.g., targeted courier incentives or pricing changes). Define: experimental unit and randomization (e.g., ZIP clusters within LA with interference mitigation), primary and guardrail metrics, success criteria, power/MDE assumptions, sample size, duration, and a ramp plan. Address seasonality and spillovers explicitly. C) If an RCT is infeasible, specify a quasi-experimental alternative (e.g., difference-in-differences or synthetic control using other SoCal metros), including pre-trend checks and robustness tests. Provide a concrete analysis checklist for either path so that another analyst could execute it without ambiguity.

Quick Answer: This question evaluates a data scientist's competency in funnel-based diagnostics, causal inference, and experimental design for distinguishing supply-side versus demand-side drivers in a two-sided marketplace, emphasizing metric selection, segmentation, and guardrail monitoring.

|Home/Analytics & Experimentation/DoorDash
DoorDash logo
DoorDash
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
12
0

LA Dinner-Period Orders Down 12% WoW: Diagnose and Validate Root Cause

Context

You are analyzing a weekly decline in a two-sided delivery marketplace. For the week of 2025-08-25 vs. week of 2025-08-18 in Los Angeles:

  • Completed orders: −12% WoW
  • Early indicators: sessions +4% WoW, order creations +1% WoW
  • Accept rate: 94% → 88%
  • p95 ETA: 42 → 58 minutes
  • Cancellation (reason = "no_courier"): 1.5% → 5.8%
  • Restaurant availability, pricing, and marketing spend: unchanged
  • Most of the decline occurred 18:00–21:00 local time

Task

Design a rigorous investigation and test plan that cleanly separates demand-side from supply-side causes and validates the true driver.

A) Diagnosis

Lay out a funnel from session → order_created → accepted → picked_up → completed. Specify:

  1. Exact cuts: by hour, ZIP, courier density, restaurant category, distance bands, and surge level.
  2. Sanity checks for tracking/config regressions.
  3. The minimal set of plots/tables to prove whether supply is the bottleneck.
  4. Additional metrics you would compute, and thresholds that would convince you.

B) Causal Validation (Experiment)

Assuming supply shortage during the dinner peak is your leading hypothesis, propose an experiment to fix it (e.g., targeted courier incentives or pricing changes). Define:

  • Experimental unit and randomization (e.g., ZIP clusters within LA with interference mitigation)
  • Primary and guardrail metrics
  • Success criteria
  • Power/MDE assumptions, sample size, and duration
  • Ramp plan
  • How you will address seasonality and spillovers

C) Quasi-Experimental Alternative

If an RCT is infeasible, specify a quasi-experimental alternative (e.g., difference-in-differences or synthetic control using other SoCal metros), including pre-trend checks and robustness tests. Provide a concrete analysis checklist for either path so another analyst can execute without ambiguity.

Loading comments...