# Evaluate Subjective and Nondeterministic Agent Outputs Design an evaluation program for an AI agent whose answers may be valid in several different forms, vary across repeated runs, and require subjective judgment. Direct user feedback is sparse. An LLM-based judge may be overconfident, and both evaluator prompts and developer-written rubrics may encode bias. Explain how you would evaluate changes before release, monitor quality online, detect evaluator failures, and decide whether a new version should ship. Include task construction, rubrics, repeated sampling, human review, user signals, statistical analysis, and safeguards against optimizing only for the chosen evaluator. ### Constraints & Assumptions - The agent performs multiple task types, so one aggregate score may hide regressions. - Some failures are rare but severe. - Production logs must be sampled with privacy and security controls. - Evaluation cost is limited, so not every output can receive expert review. ### Clarifying Questions to Ask - Which tasks, users, and failure modes matter most? - What does success mean beyond user engagement? - How much run-to-run variance does the agent exhibit? - Which release decisions require statistical confidence or safety sign-off? ### What a Strong Answer Covers - A stratified, versioned evaluation set with clear task-specific rubrics - Multiple samples per task and uncertainty-aware comparisons - Calibrated human review and measured judge agreement rather than blind trust - Online behavioral signals with confounding and harm caveats - Slice analysis, severe-failure gates, evaluator audits, and rollback criteria ### Follow-up Questions - How would you detect reward hacking against the LLM judge? - What if offline quality rises while user task completion falls? - How would you compare two agents when outcomes are paired but stochastic? - How can evaluator bias be measured across languages or user groups?