You are a Product/Data Scientist at DoorDash.
A key metric “# of successful orders per day” in Los Angeles (LA) has dropped noticeably over the last few days/weeks.
Assume:
-
A
successful order
= an order that is placed and ultimately delivered (not canceled) within the observation window.
-
You have access to event logs and operational tables for customers, dashers, restaurants, and orders.
Answer the following:
-
How would you investigate
the metric drop end-to-end? Outline your approach, the first checks you’d run, and how you’d narrow to root causes.
-
Give
one plausible hypothesis each
from the perspectives of:
-
Dasher
(supply/fulfillment side)
-
Customer
(demand/conversion side)
-
Restaurant
(merchant/operations side)
-
Suppose
all three hypotheses contribute
to the drop. How would you
quantify each hypothesis’s impact
(i.e., estimate how much each contributes to the overall decrease)? Be specific about the method, required data, and assumptions.
-
If you investigate via a
funnel
, define a reasonable funnel for DoorDash orders in LA, list key drop-off points, and propose additional hypotheses.
-
After proposing potential improvements, explain
how you would design an A/B test
to validate an improvement (choose an example improvement). Include:
-
Unit of randomization and why
-
Primary metric + guardrails
-
Power/MDE considerations
-
Key pitfalls (e.g., interference/network effects, seasonality) and how you’d handle them