Design a Multi-Agent Document Search and Visualization Product

Quick Overview

An ML system-design interview about building a multi-agent product for document search, evidence retrieval, and visualization. Candidates must define agent responsibilities, coordinate tool use, preserve citations, manage failures, and evaluate the product end to end.

Design a Multi-Agent Document Search and Visualization Product

Company: Hebbia

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Onsite

# Design a Multi-Agent Document Search and Visualization Product Design a document-analysis product that searches PDF content, extracts figures and tables, produces structured tables and charts, and streams an answer. The proposed workflow has a status agent, an orchestration agent, specialized search agents, and a final streaming agent. ### Constraints & Assumptions - Documents may contain scanned pages, embedded text, figures, and tables. - Every extracted value must retain page-level provenance. - The UI needs progress updates and partial results without presenting unverified values as final. - Agent retries must not duplicate expensive extraction jobs. ### Clarifying Questions to Ask - What user question types and document sizes must be supported? - Which outputs require exact extraction and which allow semantic retrieval? - What latency budget applies to first status, first evidence, and final answer? ### Part 1 — Architecture Define the responsibilities and contracts of orchestration, retrieval, extraction, status, and answer components. Explain whether each component must be an agent. #### What This Part Should Cover - Typed task and evidence schemas - Deterministic tools for OCR, table parsing, and charting - Durable job state and idempotent retries ### Part 2 — Quality and deployment Design evaluation, provenance, human-review thresholds, streaming behavior, and deployment monitoring from frontend through backend. #### What This Part Should Cover - Golden document-question sets - Cell- and citation-level accuracy - Backpressure, cancellation, cost, and failure observability ### What a Strong Answer Covers - A reasoned agent-versus-tool boundary - End-to-end provenance - Safe partial-result and retry semantics ```hint Prefer typed evidence over agent prose Have retrieval and extraction components return structured evidence with document, page, region, method, and confidence. The answer layer should consume that record rather than untraceable text. ``` ### Follow-up Questions - How would you handle a chart whose axes are visible but data labels are not? - How would you prevent one slow document from blocking the whole answer?

Quick Answer: An ML system-design interview about building a multi-agent product for document search, evidence retrieval, and visualization. Candidates must define agent responsibilities, coordinate tool use, preserve citations, manage failures, and evaluate the product end to end.

|Home/ML System Design/Hebbia
Hebbia logo
Hebbia
Jun 6, 2026, 12:00 AM
hardMachine Learning EngineerOnsiteML System Design
0
0

Design a Multi-Agent Document Search and Visualization Product

Design a document-analysis product that searches PDF content, extracts figures and tables, produces structured tables and charts, and streams an answer. The proposed workflow has a status agent, an orchestration agent, specialized search agents, and a final streaming agent.

Constraints & Assumptions

  • Documents may contain scanned pages, embedded text, figures, and tables.
  • Every extracted value must retain page-level provenance.
  • The UI needs progress updates and partial results without presenting unverified values as final.
  • Agent retries must not duplicate expensive extraction jobs.

Clarifying Questions to Ask Guidance

  • What user question types and document sizes must be supported?
  • Which outputs require exact extraction and which allow semantic retrieval?
  • What latency budget applies to first status, first evidence, and final answer?

Part 1 — Architecture

Define the responsibilities and contracts of orchestration, retrieval, extraction, status, and answer components. Explain whether each component must be an agent.

What This Part Should Cover Guidance

  • Typed task and evidence schemas
  • Deterministic tools for OCR, table parsing, and charting
  • Durable job state and idempotent retries

Part 2 — Quality and deployment

Design evaluation, provenance, human-review thresholds, streaming behavior, and deployment monitoring from frontend through backend.

What This Part Should Cover Guidance

  • Golden document-question sets
  • Cell- and citation-level accuracy
  • Backpressure, cancellation, cost, and failure observability

What a Strong Answer Covers Guidance

  • A reasoned agent-versus-tool boundary
  • End-to-end provenance
  • Safe partial-result and retry semantics

Follow-up Questions Guidance

  • How would you handle a chart whose axes are visible but data labels are not?
  • How would you prevent one slow document from blocking the whole answer?

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...