How would you test product changes?
Company: DoorDash
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
You are interviewing for a Product Data Scientist role at a food-delivery marketplace.
Answer the following related experimentation cases:
1. **Checkout credit default**
The app currently auto-applies any available customer credit at checkout. Product wants to change this default behavior. How would you evaluate whether the change is good or bad?
- State the product hypothesis and define the treatment and control.
- Choose the experimental unit and explain whether the test should be triggered only for users who have available credit.
- Define a primary success metric and several guardrail metrics.
- Discuss segmentation, test duration, power, and likely sources of bias or contamination.
2. **Duplicate sponsored listing in search**
In search results, if the organically ranked number-one restaurant is also the sponsored restaurant, the user may see the same restaurant twice. Is that good or bad for the marketplace? If the team wants to deduplicate the results, how would you test the change?
- Balance user experience, ad revenue, merchant fairness, and downstream order conversion.
- Define the randomization unit, success metrics, and major failure modes.
- Explain how you would reason about tradeoffs if user metrics improve but advertising metrics decline.
3. **Restaurant promotions**
Restaurants can currently launch a fixed promotion such as "$5 off $30".
- How would you increase restaurant adoption of promotions?
- The team is considering customizable promotions such as "$3 off $15" or "$10 off $50". How would you design and test this feature?
- If you had to compare the new customizable-promotion option against the existing fixed "$5 off $30" option, how would you run the experiment and decide which is better?
- Address merchant adoption, customer conversion, incremental demand, cannibalization, discount cost, and both merchant and platform profitability.
Your answer should explicitly account for the fact that this is a multi-sided marketplace with customers, restaurants, and advertising revenue.
Quick Answer: This question evaluates experimental-design and analytics competencies—specifically A/B testing, causal inference, metric selection, segmentation, power analysis, and trade-off assessment across customers, restaurants, and advertising stakeholders in a multi-sided marketplace.