North-Star and Guardrail Metrics for Google Search Result Quality

How would you tell whether Google's search results are doing their job? Define what success means for those results and how you would measure it. ```hint Watch what happens after the results load Think about what a searcher does in the moments after the results appear when the results worked, and what they do when the results did not. ``` ### Clarifying Questions - Should we measure only the ranked web results, or the whole results page, including ads, direct answers and other modules? - Is the goal to track overall search quality over time, or to judge whether a specific ranking change is an improvement? - Should success be defined differently for navigational queries, quick factual questions and open-ended research? - Do we include cases where there may be no click at all, such as answers shown directly on the results page? ### What a Strong Answer Covers - A definition of success from the searcher's point of view and a north-star metric built on it - Behavioral signals that separate satisfied from unsatisfied searches, including how searches with no click are treated - Offline relevance judgments from human raters used alongside live behavioral metrics - Guardrails such as latency, ad load and long-term return usage, with results broken down by query type - Why raw query volume or click-through rate on its own can mislead ### Follow-up Questions - Searches per user rose after a ranking change. Is that good or bad, and how would you find out? - How would you measure success for a query answered directly on the results page, where nobody clicks? - How would you catch a change that helps popular queries but hurts rare, long-tail ones?