# Design an Agent Harness and Evaluation System Design a harness and evaluation platform for a tool-using AI agent. The agent receives a user task, may take multiple model and tool steps, and returns a final result. The platform must support repeatable offline evaluation, safe execution, failure diagnosis, and controlled comparison of agent versions. Focus on the harness and evaluation design rather than model training or low-level model serving. State a representative task domain and make assumptions explicit. ### Constraints & Assumptions - Tools may be stateful, slow, unavailable, or capable of destructive actions. - Some outcomes can be checked deterministically; others need rubric-based or human review. - Model outputs are stochastic and costs vary by trajectory length. - Evaluation data may contain sensitive information and must not leak into prompts or logs unnecessarily. - A candidate version must not mutate production state during offline evaluation. ### Clarifying Questions to Ask - What agent tasks and failure costs matter most? - Which tools and permissions exist, and can they be simulated or replayed? - What is the release decision the evaluation must support? - Which dimensions matter besides task success, such as latency, cost, safety, and user effort? ### Hints - Separate task specification, execution, observation, grading, and reporting. - Preserve enough trajectory data to reproduce failures without storing secrets indiscriminately. - Treat evaluator reliability as something that must itself be measured. ### What a Strong Answer Covers - A clear harness lifecycle, isolation boundary, tool contract, and trace schema. - A representative and versioned evaluation set with deterministic and judgment-based graders. - Metrics, repeated trials, uncertainty, regression analysis, and release gates. - Safety, privacy, contamination, flaky-tool, and evaluator-bias controls. - An operational loop that turns failures into actionable categories and new tests. ### Follow-up Questions 1. How would you evaluate long-horizon tasks whose environment changes over time? 2. How do you prevent a model-based grader from favoring the model family it resembles? 3. What should be replayed, mocked, or executed live? 4. How would you detect reward hacking against the evaluation suite?