Defining a North-Star Metric and Guardrails for Web Search Inside ChatGPT

You are a product manager at OpenAI. How would you tell whether search in ChatGPT is succeeding? Lay out the metrics you would track. ```hint Clicks mean something different here In a classic search engine, a click on a result is the main signal of success. Think about what a click, or the absence of one, tells you when the product writes the answer itself. ``` ### Clarifying Questions - Is this the search capability inside ChatGPT conversations, or ChatGPT used as a browser's default search engine? - Has the feature already launched, so we are judging an existing product, or are we setting success criteria before launch? - What is the main goal: keeping ChatGPT's answers current and trustworthy, or winning queries that people take to traditional search engines today? - Should free and paid users be measured together or separately? ### What a Strong Answer Covers - The user and business goals of search inside ChatGPT, and how they differ from those of a traditional search engine. - A north-star metric that combines reach with answer quality, with a clear reason it beats click-through or thumbs-up rate. - Supporting metrics along the journey: when search is triggered, answer accuracy and citation quality, speed, in-the-moment satisfaction, and repeat use. - Guardrails such as cost per query, latency, unsupported claims, and effects on conversations that do not need search. - How the metrics would be measured credibly: human evaluation, experiments and holdouts, and segmentation by query type. ### Follow-up Questions - Thumbs-up rate on search answers rises while clicks to cited sources fall. Is that good or bad, and what would you check? - The team wants search to trigger automatically on more queries. How would you decide whether to ship that change? - Weekly search users fall a few percent week over week. How would you diagnose it?