Choose Precision, Recall, or F1 for Agent Evaluation

You need to evaluate three agents whose outputs have different levels of trustworthiness. For a particular task, each agent can make a positive decision, such as flagging an item as requiring action. Ground-truth labels can be obtained for a representative evaluation set. How would you choose among precision, recall, and F1 when evaluating these agents? Explain how your choice changes when a false positive is more costly than a missed positive, when a missed positive is more costly, and when both error types matter. Do not assume that an agent's reputation determines which metric should be used for it. ### Constraints and Clarifying Questions - Define the positive class and the action taken on a positive prediction before choosing a metric. - Compare agents on the same task, labeling standard, and relevant case distribution. - Clarify the cost of a false positive and a false negative, and whether outputs are reviewed by a person before action. - State whether the agents expose confidence scores that support threshold selection or only final decisions. ```hint Start from a wrong decision For the same item, consider the consequence of acting when no action was needed and the consequence of failing to act when it was needed. A single aggregate score can conceal either failure. ``` ### What a Strong Answer Covers - Definitions of precision, recall, and F1 using true positives, false positives, and false negatives. - A metric choice tied to the consequences of each error type rather than a vague trust ranking. - A fair comparison of the three agents, including thresholds when available and the underlying confusion counts. - The limitations of F1 and of reporting a metric on a small or unrepresentative sample. ### Follow-up Questions - How would you compare a cautious agent that rarely produces a positive decision with one that flags most items? - What should you report when an agent predicts no positives or the evaluation set contains no actual positives? - If an agent's positive decisions are automatically executed instead of reviewed by a person, how might your evaluation criterion change?