Launch Decision for a More Accurate Model That Is Twice as Slow

You are the product manager for a feature powered by a machine-learning model. The team brings you a candidate model that is 20% more accurate than the model now in production, but its latency is twice as high. Would you ship it? Walk the interviewer through how you reach the decision. ```hint Price both numbers for the user Convert the accuracy gain and the extra wait into what a user of this feature actually experiences, and ask whether the decision really has to apply to every request in the same way. ``` ### Clarifying Questions - Which product and surface does the model power: an interactive experience such as search, autocomplete or chat where people wait for the result, or a background task such as classification, ranking or batch summaries? - Is the 20% a relative gain or 20 percentage points, on which metric (error rate, precision on a key class, human-rated quality), and was it measured offline or with real users? - What are the current latency figures in absolute terms, and is "twice as high" measured at the median, at the slow tail, or as time to the first visible output? - Does the slower model also cost more to serve, and does this surface have a fixed latency budget or service commitment? ### What a Strong Answer Covers - Translating both numbers into impact on this surface: the baseline behind the 20%, where the gain lands, and whether the absolute added delay is noticeable - Validating the offline gain with an online experiment that measures the accuracy and latency effects directly - Options between shipping and holding, such as routing requests, reducing latency, changing the interface or a targeted rollout - Guardrails and costs: serving cost and capacity, slow-tail latency and timeouts, abandonment, and segments such as users on slow networks - A committed recommendation with the results that would change it ### Follow-up Questions - The online test shows higher task success but a small, steady decline in sessions per user over a month. What do you recommend? - Engineering says it can cut the new model's latency substantially within two quarters. Do you wait, ship now, or ship to part of the traffic? - How would your reasoning change if the model supported a medical or financial decision instead of a consumer feature?