Conservative Wait Estimates Cut Conversion and Pickup Time: Did Efficiency Improve?

This free-response case study comes from a timed online assessment for a data science role at a ride service. The team launched **Smart Wait**, a feature that shows riders a conservative estimate of their wait: the displayed time to pickup (TTP), the time until a vehicle reaches the rider, is deliberately on the high side of the prediction. After the launch: - conversion dropped by 5%; - the actual TTP dropped by 25%. ### Clarifying Questions - How is conversion defined: the share of riders who request a ride after seeing the estimate, or something else? - Is the 5% drop relative (for example from 40% to 38%) or in percentage points? - Is actual TTP measured over all requests, or only over rides that were completed? Are cancelled requests included? - Were these numbers produced by a randomized experiment, or by comparing periods before and after the launch? - Did anything else change over the same period, such as fleet size, dispatch logic, service area, or season? ### Part 1 — Fleet efficiency The product manager concludes that Smart Wait improved fleet efficiency. Do you agree? Explain why or why not, and say what evidence would settle it. ```hint Who is left in the data Ask which riders are still requesting rides after the displayed estimate changed, and where they are relative to the vehicles. ``` #### What This Part Should Cover - Whether lower TTP measures efficiency at all, or only the mix of requests that remain - The link between fewer requests, idle vehicles and shorter pickups - Metrics that do measure fleet efficiency, and a comparison design that isolates the feature's effect ### Part 2 — Other explanations What other hypotheses could explain the drop in actual TTP? For each one, explain how it relates to the drop in conversion, and how you would check it with data. ```hint Sort by mechanism Group hypotheses by whether they would also move conversion, or would lower TTP on their own. ``` #### What This Part Should Cover - Selection and composition hypotheses caused by the displayed estimate - Supply-side and external hypotheses that are unrelated to the feature - Measurement hypotheses (how TTP and conversion are logged and defined) - For each hypothesis, the predicted relationship to conversion and the data cut that tests it ### What a Strong Answer Covers - Treating the two metric movements as possibly causally linked rather than as two independent results - Recognizing composition effects that make an average improve while no individual ride gets faster - Concrete fleet efficiency metrics rather than TTP alone - An analysis or experiment design that separates the feature's effect from other changes - A recommendation that weighs the lost conversion against any genuine efficiency gain ### Follow-up Questions - How would you design an experiment for Smart Wait, given that treated and untreated riders share the same vehicles? - The team wants to keep the feature but recover conversion. What would you test? - Which single metric would you put on the launch dashboard as the guardrail, and why?