Uber ETA Feature: Choosing a Primary Success Metric, Accuracy Measures and Guardrails

You are a product manager at Uber responsible for designing the estimated time of arrival (ETA) that the app shows its customers. How would you measure whether that ETA design is a success? Explain what the ETA is for, choose the metric you would treat as the main measure of success, and lay out the supporting and guardrail metrics around it. ```hint Useful versus merely accurate An estimate can look good on an accuracy chart and still fail the person reading it. Think about what riders and drivers do differently because of the ETA, and how a team could make an accuracy number look better without helping anyone. ``` ### Clarifying Questions - Which estimate is in scope: when the driver will reach the pickup point, when the trip will reach the destination, or both? And is it the estimate shown before the rider requests, after a driver is matched, or both? - Is this the rides product only, or should delivery ETAs in Uber Eats be included? - Are we judging a new prediction model, a change to how the ETA is displayed, or the ETA experience as a whole? - Should the metrics cover drivers as well as riders? ### What a Strong Answer Covers - A clear statement of what the ETA is for, for riders, drivers and the marketplace, tied to the specific points in the trip where it appears. - One primary success metric with a precise, justified definition, rather than an unranked list. - Accuracy measures that capture bias, tail errors and how much the estimate changes, not only an average error. - Behavioral and business metrics linked to the ETA, such as request conversion, cancellations and contacts, with attribution handled carefully. - Guardrails against gaming such as padded estimates, segment cuts, and a sound way to experiment in a two-sided marketplace. ### Follow-up Questions - A new ETA model lowers average error, yet riders cancel more often after being matched. What might explain it, and what would you do? - Should the app show a single arrival time or a range, and how would you test which works better? - How would you run an experiment on a new ETA model when riders and drivers in the same city affect each other?