Designing a Marketplace-Safe Experiment for a New Uber Driver Feature

Uber wants to test a new feature built for its drivers. As the product manager, how would you design an A/B test that gives a trustworthy read on the feature without harming the platform, meaning riders, drivers and the business, while the test is running? The feature itself is not specified. You may pick a concrete example to make your design specific, but say which one you chose. ```hint Drivers share the same trips Drivers in a city compete for the same pool of ride requests. Before choosing what to randomize, think about what that competition does to a test that puts some drivers in treatment and their neighbors in control. ``` ### Clarifying Questions - What does the feature do: does it change which trips drivers accept, where they wait, or how long they stay online, or does it only change what they see in the app? - What decision should the test inform: launch or not, which of several variants, or whether to invest further? - Which cities are candidates for testing, and are any off limits, for example places with strict rules on how driver pay is set? - What harm would leadership treat as unacceptable during the test: longer rider waits, lower driver earnings, or something else? ### What a Strong Answer Covers - A primary metric tied to the feature's goal, plus guardrails on both sides of the marketplace (rider waits and fulfillment, driver earnings and retention) - Recognition that drivers compete for shared demand, so randomizing individual drivers can bias the result, and a deliberate choice of unit (driver, market, region and time window) with its trade-offs - Sample size, duration and ramp plan that reflect weekly demand cycles, novelty effects and the small number of market-level units - Risk controls during the test: limited initial exposure, pre-agreed stop rules, a kill switch and fairness to drivers in both groups - How the results are read, including effects on drivers who did not get the feature, and turned into a launch decision ### Follow-up Questions - The test shows treated drivers earned more per hour, while control drivers in the same cities earned less. How do you interpret that, and do you launch? - The feature is a setting drivers configure once and keep, so it cannot be switched on and off by time window. How would you test it instead? - How would you tell a lasting effect apart from drivers trying something new during the first week?