Design a Spam Review Workflow with Classifiers, AI Agents, and Human Labelers

# Design a Spam Review Workflow with Classifiers, AI Agents, and Human Labelers Design a spam detection system that combines a classifier, an AI agent, and human labelers. The main objective is to improve the accuracy and efficiency of human review. Explain the responsibilities of each component, the routing of cases, and how you would evaluate the workflow. Discuss useful AI capabilities and their limitations without assuming an agent's output is reliable ground truth. ### What a Strong Answer Covers - A workflow that reserves human attention for appropriate cases and preserves a path for escalation. - A bounded role for AI assistance and reliable label-quality controls. - Measures of review accuracy and throughput that account for case difficulty and selection. - Evaluation of automation errors, reviewer overreliance, and distribution changes. ```hint Separate assistance from authority Consider which evidence an agent can organize and which consequential decisions still require verified rules or human judgment. ``` ### Follow-up Questions - How would you detect whether the agent makes reviewers faster but less accurate? - How would you learn about spam that the classifier never sends to review?