Measuring a Live Chatbot's Tool-Call Errors, Hallucinations and Unsafe Replies

Your team at Google runs a chatbot that is already live with real users. How would you get a clearer picture of how it actually behaves in production, especially when it calls tools, when it hallucinates, and when it gives inappropriate responses? ```hint Look past the test set Offline benchmarks contain only the cases someone thought to write. Think about what evidence live traffic itself can give you, and how each of the three problem areas would leave a different trace in it. ``` ### Clarifying Questions - What should this understanding serve first: catching regressions after model updates, prioritizing quality work, or reporting risk to leadership and regulators? - Which tools can the chatbot call (search, code execution, actions in other apps), and can any of them change something in the outside world? - What privacy rules govern looking at real conversations: may people read sampled chats, and under which consent and retention terms? - Is there already a written policy that defines an inappropriate response, or does that definition need to be set? ### What a Strong Answer Covers - Measurable definitions of tool-use failures, hallucinations and inappropriate responses, with severity levels. - A mix of signals (production logs, implicit and explicit user feedback, automated raters, sampled human review) and how they are calibrated against each other. - Metrics sliced by intent, language, tool and model version, plus a way to discover failure modes nobody anticipated. - The privacy and access safeguards the measurement itself needs. - How findings turn into fixes: triage, regression sets built from production failures, release gates and alerts. ### Follow-up Questions - After a launch the thumbs-down rate fell, but sampled human review shows more hallucinations. Which do you trust, and why could they disagree? - How would you estimate the hallucination rate on questions whose correct answer is not yet known when you review them? - A new model version is due next week. How would you use this system to decide whether it ships?