## Evaluate an Agentic Decision System End to End You are replacing a rule-based workflow with an agent that selects tools and recommends operational decisions. A trace shows the chosen route, each tool call, and the final recommendation. Tool execution can be correct even when the recommendation is wrong. Design an evaluation program that answers whether the agent improves decision quality and efficiency. Cover offline and online evaluation, an appropriate human baseline, comparison with the existing rules, and how you would use route precision/recall, tool success rate, A/B tests, shadow mode, and manual-override data. Explain how you would distinguish a routing failure, a tool failure, and a reasoning failure, and how you would protect users while evidence is still limited. ### Constraints & Assumptions - Outcomes have different risk levels, and some consequences arrive later than the agent response. - Historical traces contain both routine cases and rare high-impact cases. - Human reviewers are available for a bounded sample, not every decision. - The existing rule-based system remains available as a baseline and fallback. ### Clarifying Questions to Ask - What is the business outcome for each decision, and when does its label become observable? - Which errors are reversible, and which require mandatory human approval? - Are historical decisions representative of the traffic the new agent will receive? - Can shadow recommendations be joined reliably to later outcomes? ```hint Separate the layers Define success independently for routing, tool execution, policy compliance, and the final decision before combining them into an end-to-end scorecard. ``` ### What a Strong Answer Covers - A labeled offline set stratified by risk, difficulty, and important edge cases, with time-based holdouts to detect drift. - Layered diagnostics plus end-to-end outcome metrics; route and tool metrics are treated as explanatory measures, not the release objective. - Blind human comparison, inter-rater agreement, and explicit comparison with the rule-based baseline. - Shadow deployment followed by a guarded experiment with predeclared quality, safety, efficiency, and override measures. - Confidence intervals, slice analysis, failure taxonomy, auditability, rollback criteria, and a plan for delayed outcomes. ### Follow-up Questions - How would you evaluate a rare decision for which the correct outcome is not known for several weeks? - What does a high manual-override rate mean if business outcomes still improve? - How would you detect that the agent is gaming a proxy metric while harming the true objective?