Design Retrieval-Augmented Answers from a Document Vault
Company: Harvey
Role: Software Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Onsite
Design a service that accepts a user's prompt, searches a large document vault, and returns a grounded answer with citations to relevant source passages. The system may use a language model and retrieval-augmented generation, but every answer must preserve document access controls and make unsupported claims detectable.
Cover ingestion, document parsing, chunking, indexing, retrieval, ranking, prompt assembly, generation, citation mapping, freshness, and evaluation. Explain what happens when evidence is weak or conflicting.
### Constraints & Assumptions
- Documents have tenant and document-level permissions.
- The vault changes over time; updates and deletions must propagate to search.
- Documents may be long and may contain repeated or contradictory passages.
- Model context is limited, and model calls can fail or produce ungrounded text.
- Users need citations that resolve back to the exact permitted document version and passage.
### Clarifying Questions to Ask
- Which document formats, languages, and query types must be supported?
- What latency, freshness, and answer-quality targets matter most?
- May any data leave the trust boundary for embedding or generation?
- Should the system answer from partial evidence or explicitly abstain?
### What a Strong Answer Covers
- Builds a versioned ingestion pipeline and keeps chunk provenance through every stage.
- Applies authorization before or during retrieval, not only after generation.
- Combines lexical and semantic retrieval with ranking and diversity controls.
- Generates from bounded evidence and maps citations to exact source spans.
- Handles deletion, stale indexes, conflicting sources, prompt injection, and model failure.
- Defines offline retrieval and answer metrics plus online feedback and observability.
### Follow-up Questions
1. How would you prevent instructions embedded in a retrieved document from controlling the model?
2. How would you debug a correct source that was retrieved but not cited?
3. What would you cache, and how would document-version changes invalidate it safely?
Quick Answer: Design a document-vault question-answering service that returns grounded responses with citations while preserving tenant and document permissions. Cover versioned ingestion, retrieval quality, exact provenance, deletion and freshness, conflicting evidence, prompt injection, abstention, and evaluation.