Design a Rule-Guided Document Evaluation Agent
Company: Harvey
Role: AI Agent
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
## Interview Prompt
Design a system that evaluates an uploaded document against a rule set, assigns
one of three outcomes—qualified, acceptable with changes, or unqualified—and
produces concrete revision suggestions for the latter two outcomes. Extend the
design to multiple specialized agents and explain how the system prevents their
edits or judgments from conflicting.
### Constraints & Assumptions
- Every decision and suggestion must be traceable to a versioned rule and source span.
- Deterministic rules should not be delegated to a generative model unnecessarily.
- Agents may propose overlapping or contradictory document changes.
- A human reviewer must be able to inspect and override the final result.
### Clarifying Questions to Ask
- Which rules are hard eligibility gates versus advisory quality checks?
- May the system edit the document automatically, or only propose patches?
- What accuracy, latency, and audit requirements apply to each outcome?
### What a Strong Answer Covers
- A typed rule engine and extraction layer before any model-based interpretation.
- Structured findings with rule IDs, evidence spans, confidence, and proposed patches.
- A conflict-resolution stage with one authoritative editor or deterministic merge policy.
- Evaluation datasets covering false qualification, missed issues, and conflicting suggestions.
- Versioning, human review, observability, and replay of past decisions under new rules.
### Follow-up Questions
- How would you detect that two individually valid edits conflict when combined?
- When should the system abstain rather than choose an outcome?
- How would rule updates affect documents evaluated under an older version?
Quick Answer: Design a rule-guided agent that classifies uploaded documents and produces traceable revision suggestions. Separate deterministic rules from model judgment, bind every finding to rule versions and source spans, reconcile conflicts among specialist agents, and preserve human override.