Design a Rule-Guided Document Evaluation Agent

Quick Overview

Design a rule-guided agent that classifies uploaded documents and produces traceable revision suggestions. Separate deterministic rules from model judgment, bind every finding to rule versions and source spans, reconcile conflicts among specialist agents, and preserve human override.

Design a Rule-Guided Document Evaluation Agent

Company: Harvey

Role: AI Agent

Category: ML System Design

Difficulty: medium

Interview Round: Onsite

## Interview Prompt Design a system that evaluates an uploaded document against a rule set, assigns one of three outcomes—qualified, acceptable with changes, or unqualified—and produces concrete revision suggestions for the latter two outcomes. Extend the design to multiple specialized agents and explain how the system prevents their edits or judgments from conflicting. ### Constraints & Assumptions - Every decision and suggestion must be traceable to a versioned rule and source span. - Deterministic rules should not be delegated to a generative model unnecessarily. - Agents may propose overlapping or contradictory document changes. - A human reviewer must be able to inspect and override the final result. ### Clarifying Questions to Ask - Which rules are hard eligibility gates versus advisory quality checks? - May the system edit the document automatically, or only propose patches? - What accuracy, latency, and audit requirements apply to each outcome? ### What a Strong Answer Covers - A typed rule engine and extraction layer before any model-based interpretation. - Structured findings with rule IDs, evidence spans, confidence, and proposed patches. - A conflict-resolution stage with one authoritative editor or deterministic merge policy. - Evaluation datasets covering false qualification, missed issues, and conflicting suggestions. - Versioning, human review, observability, and replay of past decisions under new rules. ### Follow-up Questions - How would you detect that two individually valid edits conflict when combined? - When should the system abstain rather than choose an outcome? - How would rule updates affect documents evaluated under an older version?

Quick Answer: Design a rule-guided agent that classifies uploaded documents and produces traceable revision suggestions. Separate deterministic rules from model judgment, bind every finding to rule versions and source spans, reconcile conflicts among specialist agents, and preserve human override.

|Home/ML System Design/Harvey
Harvey logo
Harvey
Aug 14, 2026
mediumAI AgentOnsiteML System Design
4
0

Interview Prompt

Design a system that evaluates an uploaded document against a rule set, assigns one of three outcomes—qualified, acceptable with changes, or unqualified—and produces concrete revision suggestions for the latter two outcomes. Extend the design to multiple specialized agents and explain how the system prevents their edits or judgments from conflicting.

Constraints & Assumptions

  • Every decision and suggestion must be traceable to a versioned rule and source span.
  • Deterministic rules should not be delegated to a generative model unnecessarily.
  • Agents may propose overlapping or contradictory document changes.
  • A human reviewer must be able to inspect and override the final result.

Clarifying Questions to Ask Guidance

  • Which rules are hard eligibility gates versus advisory quality checks?
  • May the system edit the document automatically, or only propose patches?
  • What accuracy, latency, and audit requirements apply to each outcome?

What a Strong Answer Covers Guidance

  • A typed rule engine and extraction layer before any model-based interpretation.
  • Structured findings with rule IDs, evidence spans, confidence, and proposed patches.
  • A conflict-resolution stage with one authoritative editor or deterministic merge policy.
  • Evaluation datasets covering false qualification, missed issues, and conflicting suggestions.
  • Versioning, human review, observability, and replay of past decisions under new rules.

Follow-up Questions Guidance

  • How would you detect that two individually valid edits conflict when combined?
  • When should the system abstain rather than choose an outcome?
  • How would rule updates affect documents evaluated under an older version?

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...