Pre-Launch Testing and Live Monitoring Plan for an Autonomous Support-Ticket AI Agent

You are the product manager for an AI agent that handles customer support tickets end to end, resolving them without a human agent stepping in. Design the framework you would use to evaluate it, in two phases. ### Clarifying Questions - Which kinds of tickets is the agent expected to resolve by itself (order status, account access, billing disputes, technical troubleshooting), and can it take actions such as issuing refunds or changing account settings, or does it only answer questions? - Is there an existing human support team whose past tickets and resolutions can serve as a baseline and as labeled test data? - When the agent is unsure or the customer asks for a person, does it hand the ticket to a human, and should that handoff count as a failure? - Which failures does the business treat as unacceptable at any rate, for example a refund issued in error or one customer's data shown to another? ### Part 1 — Before launch What does the agent have to demonstrate before it is allowed to close real customers' tickets on its own, and how would you test each of those points? ```hint Grade what happened, not how it sounded A polite, fluent reply can still leave the customer's problem unsolved or trigger the wrong account change. Think about what the agent's behavior on each test ticket should be compared against. ``` #### What This Part Should Cover - How the test set is assembled and what each test ticket is graded against - Which quality dimensions are graded separately, and who or what does the grading - Staged exposure to real tickets and the bar that must be met before each step ### Part 2 — After launch Once the agent is resolving live tickets, how do you know it is really solving customers' problems, and how do you catch it when quality slips? ```hint Closed is not the same as resolved A ticket the agent closes today may be reopened, escalated or followed by another contact about the same issue next week. Decide when a ticket counts as truly resolved. ``` #### What This Part Should Cover - A primary production metric based on true resolution, with supporting metrics and guardrails - A fair comparison with human-handled tickets that is not distorted by which tickets the agent receives - Ongoing monitoring, regression checks on every change, and how production failures flow back into testing ### What a Strong Answer Covers - One definition of a successful resolution, used consistently before and after launch - Different evaluation bars by ticket type and action risk, with the strictest bar where a mistake costs money, exposes data or breaks policy - Customer outcomes (satisfaction, effort) weighed alongside business outcomes (cost per ticket, load on human agents) - Explicit rules and owners for expanding, pausing or rolling back the agent's autonomy ### Follow-up Questions - The agent's resolution rate looks strong, but satisfaction scores on its tickets trail those on human-handled tickets. How do you investigate? - The underlying model is upgraded to a new version. What has to happen before the agent runs on it in production? - How would you evaluate the agent on a type of ticket that did not exist when you built the pre-launch test set?