A/B Testing Short-Distance Bike Courier Dispatch in a Food-Delivery Marketplace

A food-delivery marketplace matches customers' orders from merchants with couriers who pick them up and deliver them. In an analytics case, you are asked to walk through the full A/B testing process for a change involving delivery by bike: the hypothesis, the metrics, how you would randomize, how large and how long the test should be, how you would analyze the results, and how you would decide whether to launch. The report identifies the case only by its subject, bikes, and says it tested the A/B testing process. This version fixes the change so the design can be concrete: in dense city zones, the dispatch system would start offering short-distance orders to couriers who deliver by bicycle or e-bike, alongside couriers who drive. The company wants to know whether to roll this out to more zones. ```hint Shared orders Bike couriers and drivers take orders from the same pool of nearby orders. Ask what happens to control-group orders when the treatment changes who picks up treatment-group orders. ``` ```hint What can be assigned Couriers decide for themselves whether to ride a bike. Ask which unit you can actually assign to treatment at random, and what that unit costs you in sample size. ``` ### Clarifying Questions - What is the main goal: faster deliveries, more courier capacity at busy times, lower cost per delivery, or attracting couriers who do not have a car? - Are bike couriers already active in these zones, or would the change bring in new ones? - How is "short distance" defined, and is the threshold fixed or tuned during the test? - How many zones are eligible, and can the dispatch rule be switched on and off by zone and by time of day? - Which quality and safety measures does the company already treat as guardrails? ### What a Strong Answer Covers - A testable hypothesis and launch decision tied to the goal, with one primary metric, secondary metrics, and guardrails for customers, couriers and merchants - A randomization unit chosen to handle marketplace interference and courier self-selection, with its cost in statistical power stated - Sizing from the variance of the chosen unit, a minimum detectable effect, and a duration that covers weekly cycles and weather - An analysis plan with sanity checks, variance reduction, correctly clustered errors and pre-planned segments - A decision rule and rollout plan, including what to do when speed and quality move in opposite directions ### Follow-up Questions - Only a handful of zones qualify. How would you still estimate the effect credibly? - Delivery times fall, but complaints about cold or spilled food rise. How do you decide? - How would you measure whether the change brings more couriers onto the platform, given that couriers choose whether to sign up with a bike? - The effect is large in the first week and then shrinks. What could explain that, and what would you do?