Measuring Quality of an AI Deep Research Agent: Eval Sets, Graders and Release Gates

You are a product manager working on Deep Research, an AI agent that investigates a user's question across many sources and writes a long report with citations. How would you build evals for it? ```hint Count the ways a report can fail A long, cited report can go wrong in several distinct ways. List them before choosing how to measure each one, and remember the agent's search steps, not only the final text. ``` ### Clarifying Questions - Is Deep Research for consumers, or for business users whose reports also draw on internal work files, which adds permission and privacy checks? - Are the evals mainly for comparing model and agent versions before release, for monitoring quality in production, or both? - Which kinds of research matter most, for example market and competitor analysis, technical literature reviews, or health and legal questions where an error is costly? - How much budget is there for expert human graders compared with automated grading? ### What a Strong Answer Covers - A breakdown of report quality into separately measured dimensions, such as factual accuracy, whether citations support their claims, coverage, synthesis and instruction following, each linked to how it hurts users - How the eval sets are built: representative questions across domains and difficulty, expert reference material, freshness, and protection against training contamination - Grading methods and their calibration: automated checks, model-based graders validated against expert judgments, and pairwise comparisons between versions - Evaluation of the agent's process (search and source quality, cost, time) alongside the final report, plus safety tests - How offline results drive release decisions and connect to online signals after launch ### Follow-up Questions - Your model-based grader scores a new version higher, but expert reviewers prefer the old one. What do you do? - How would you catch a report that cites a real page which does not say what the report claims? - Reports get longer and score better on coverage, while users read less of each one. How do you change the evals?