Judging Whether an AI Product Is Working: Metrics, Guardrails and Launch Bars

You are the product manager for an AI tool. How would you define what success looks like for it, and how would you measure it? ```hint Generated is not the same as useful Think about what a user does with the AI's output right after receiving it, and which signals would separate "the model produced something" from "the user's work actually got easier". ``` ### Clarifying Questions - Which AI tool is this, and where does it live: an assistant built into Atlassian's existing products such as Jira and Confluence, or a standalone product? - Does the AI only suggest content that a person reviews and accepts, or can it act on its own, for example by creating or editing work items? - Is the tool included for every customer or sold as a paid add-on, so that revenue is part of success? - Are these criteria for deciding whether a beta is ready for general availability, or for running the tool after launch? ### What a Strong Answer Covers - Why the tool exists for users and for the business, stated before any metric is chosen. - Separate layers of success: quality of the AI output, adoption by users, the outcome for the user's work, and business impact. - One primary metric with supporting metrics and AI-specific guardrails such as factual errors, content exposed across permissions, latency and cost per request. - How impact is attributed to the tool (holdouts or experiments) rather than to novelty or seasonality. - Concrete pass or fail bars for each stage, from beta to broad rollout. ### Follow-up Questions - Weekly usage is strong for the first two weeks and then halves. How do you tell novelty wearing off from a tool that is not useful? - Users accept most suggestions, but an audit of sampled outputs finds frequent errors. What does that say about your primary metric, and what would you change? - The serving cost per active user is higher than the price can support. Which of your criteria decides what happens next?