How to test bike delivery?
Company: DoorDash
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You are a data scientist at a food-delivery marketplace. The company is considering launching a **bicycle courier delivery option** in selected cities.
Design a data-driven approach to evaluate this launch.
Please address the following:
1. **Why might bike delivery be worth considering?** Discuss potential business, customer, courier, and operational benefits versus existing delivery modes such as cars or scooters.
2. **What factors should be considered before launching or testing it?** Include marketplace effects, city density, delivery distance, terrain, weather, order mix, courier supply, parking, safety, regulatory constraints, and possible cannibalization of existing courier modes.
3. **What metrics would you use?** Define a north-star metric, supporting operational metrics, and guardrail metrics. Explain tradeoffs among speed, cost, fulfillment, quality, courier economics, and safety.
4. **What type of experiment would you run?** Explain the most appropriate experimentation design for a two-sided marketplace, including the unit of randomization, experiment duration, segmentation, and how you would handle interference between treatment and control.
5. **How would you decide whether to scale the launch?** Describe how you would interpret results, account for confounding factors such as weather and time of day, and make a go / no-go recommendation.
Overview: This question evaluates a data scientist's competency in data-driven product evaluation, experimentation design, metric selection, causal inference, and marketplace economics within the Analytics & Experimentation domain.
Community answers
Answer by Code4Ever
Why might bike delivery be worth considering? (what's the goal of this feature?)
better dasher supply (direct) -> better delivery quality (primary) -> higher user retention and satisfaction
What factors should be considered before launching or testing it?
only launch bike in city with high population density (no rural area)
bike safety and insurance needs to be considered given it could be potentially more dangerous than cars
don't assign batch orders to bikers unless it's e-bike given it might be hard to carry multiple orders
What metrics would you use?
dasher supply (number of active dashers, number of new dasher signups)
better delivery quality
(success metrics) avg delivery time, pct of order delivered on time
(guardrail metrics) number of incidents, pct of missing and incorrect orders, pct of orders with support ticket
user retention
order rates, GOV, pct of users retained in the next 1 to 7 days
What type of experiment would you run?
geo experiment (unit of randomization is split by location that is alike but will come with low power since it's hard to find perfect matched markets)
regular experiment (suffers from network/spillover effect and impacts will be underestimated)
How would you decide whether to scale the launch?
pilot program to monitor the success metrics and guardrail metrics
run geo experiment to roughly estimate the causal impacts
scale it if everything looks good