How a PM Should Evaluate an Engineer's Proposed Overhaul of Result Ranking

An engineer on your team proposes a major change to how your product ranks results. As the PM, how would you evaluate the change and decide whether it should launch? ```hint Start from the why Before judging the new ranking, pin down what the engineer believes is wrong with today's results and for whom; that hypothesis decides which measurements matter. ``` ### Clarifying Questions - Which surface is being ranked: web search results, a content feed, ads, or app-store listings? - What problem is the change meant to fix: relevance for a class of queries, freshness, spam, latency or serving cost? - Is it a new model, new ranking signals or a new objective, and can it be switched on for only part of the traffic, such as one market or query type? ### What a Strong Answer Covers - The hypothesis behind the change and the users or query segments expected to benefit - Offline evaluation: how relevance is judged, which metrics are used, and how results are broken down by segment - Online experiment design: primary metric, guardrails, sample size, duration, and known biases in click data - Explicit launch criteria, a plan for mixed results, and a staged rollout with rollback - Second-order effects: latency and cost, content providers and advertisers, spam and feedback loops ### Follow-up Questions - The experiment shows more clicks, but users rephrase their queries more often. What do you conclude? - The change wins overall but clearly hurts results in one smaller language market. Do you launch? - How would you notice, months after launch, that the new ranking is being gamed?