Setting Launch Criteria and a Rollout Plan for an AI Agent

Your team has built an AI agent and wants to launch it. As the product manager, how would you decide whether it is ready? ```hint Good enough compared with what Before choosing metrics, pin down what users rely on today for the same task and what the worst mistake the agent could make would cost. Together those set how high the bar has to be. ``` ### Clarifying Questions - What does the agent do, and can it take actions with real consequences (sending messages, changing records, spending money), or does it only suggest actions to a person? - Who will use it: consumers, business customers or internal teams? And what do they rely on today for the same task? - Does "launch" mean release to everyone, or could it start as a limited beta, an opt-in feature, or with a person approving each action? - What evaluation assets already exist: a set of real tasks with known correct outcomes, human graders, logs from a pilot? ### What a Strong Answer Covers - A definition of "ready" tied to the agent's tasks, the baseline it replaces and the cost of its worst failures - An offline evaluation plan: representative task sets, end-to-end success, consistency across repeated runs, and how grading is done and validated - Safety, security and reliability checks specific to agents, such as irreversible actions and instructions hidden in content the agent reads - Launch criteria agreed before the results come in, plus a staged rollout with online metrics and rollback triggers - Trade-offs between capability, autonomy, cost and latency, and who signs off on the decision ### Follow-up Questions - The agent beats the human baseline on average but fails badly on a rare, high-cost type of task. Do you launch, and if so how? - How would you know that the automated grader used to score the agent can be trusted? - After launch, the provider of the underlying model releases a new version. What has to happen before you switch to it?