DoorDash Launch Call: A/B Test Raises Orders While Fewer Restaurants Receive Them

You are a product manager at DoorDash, and a new feature you own is about to launch. Its A/B test shows that users in the treatment group place more orders, but fewer distinct restaurants receive those orders: the count drops, for example, from 100 to 90. Would you launch the feature? Explain how you read the result, what you would check before deciding, and what you decide. ```hint Two sides of one marketplace Orders describe the consumer side; the restaurant count is a supply-side signal. Before trading one against the other, ask how reliably a user-split test measures restaurant outcomes and which restaurants make up the missing ones. ``` ### Clarifying Questions - What does the feature change: ranking or recommendations, a reorder shortcut, a promotion, or something else? Was it randomized by user, by market or by time window? - Are 100 and 90 the distinct restaurants ordered from by control and treatment users over the same window, and were the two groups the same size? - Is the order lift statistically significant, and does it hold for order value and repeat ordering, not only for order count? - Is anything known about the restaurants that dropped out, for example whether they are new, low-volume, independent or poorly rated? ### What a Strong Answer Covers - Treating the restaurant count as a supply-health signal and testing whether the drop is real (the sample behind a distinct count, interference between arms that share restaurants and Dashers) - Diagnosing the mechanism by segmenting the restaurants that lost orders and tracing where the extra orders went - Weighing the short-term order lift against restaurant retention, selection and consumer retention over months - A clear decision (launch, launch with guardrails, or modify and re-test) with the conditions that would change it - Concrete success and guardrail metrics for consumers, restaurants and Dashers during rollout ### Follow-up Questions - The missing restaurants turn out to be mostly ones that joined in the past month. Does your decision change? - How would you redesign the experiment so that restaurant-side effects are measured without bias? - Leadership wants the order lift this quarter. How do you argue for keeping a restaurant guardrail?