## Design an Enterprise Research Assistant with Verifiable Citations Design an enterprise research system that accepts complex questions, retrieves authorized internal and external information, synthesizes findings, attaches verifiable citations, and produces a report. The system may let a model plan steps and invoke approved retrieval or analysis tools, but every result must respect current permissions. ### Constraints & Assumptions - Internal sources have document- or finer-grained access controls that can change after ingestion or after a report is generated. - External and internal documents can be updated, deleted, duplicated, or contradictory. - Citations must identify the exact source version and evidence span supporting a claim. - Research may require several retrieval and synthesis steps and can outlive one synchronous request. - Required corpus size, latency, freshness, model budget, report length, and quality targets must be clarified. ### Clarifying Questions to Ask - Which internal connectors, external sources, file types, and languages are in scope? - What user identity and group system is authoritative for permissions? - Must results be reproducible against a source snapshot, or always reflect the newest data? - Which quality metric matters: evidence recall, claim support, report usefulness, latency, or cost? - May users share reports, and what happens when a viewer lacks access to one cited source? ### Part 1 — Define Trust and Product Contracts Specify the query and report APIs, asynchronous job states, authorization boundaries, citation behavior, and how the system responds when evidence is insufficient or conflicting. #### What This Part Should Cover - Caller identity and entitlements propagated to every tool and retrieval step. - Report, claim, evidence, citation, and source-version identities. - Abstention or uncertainty rather than unsupported completion. - Sharing rules and reauthorization when another user opens a report. ```hint Make every claim traceable A citation is useful only when the stored claim points to the exact retrieved evidence and source version used to produce it. ``` ### Part 2 — Ingest and Retrieve Internal and External Evidence Design connectors, parsing, chunking, indexing, freshness, identity resolution, hybrid retrieval, reranking, and query-time permission filtering. #### What This Part Should Cover - Replayable source versions with provenance and connector checkpoints. - Keyword and semantic indexes plus metadata filters. - Security trimming before evidence reaches the model. - Deduplication, recency, contradictory sources, and retrieval evaluation. ```hint Filter before generation Removing unauthorized text only from the final report is too late if the model already received that text in its context. ``` ### Part 3 — Plan Research and Build Cited Reports Describe how a model decomposes a complex question, invokes approved functions, tracks evidence, synthesizes claims, validates citations, and renders a report under bounded cost and time. #### What This Part Should Cover - A typed tool registry and constrained planner state. - Per-step budgets, loop limits, retries, and failure handling. - Claim-to-evidence mapping and citation entailment checks. - Human review or abstention for unsupported high-impact conclusions. ```hint Keep evidence outside free-form prose Have synthesis produce structured claims with evidence IDs before rendering the final narrative and citation markers. ``` ### Part 4 — Cache Safely and Handle Permission Revocation Design exact and semantic caching for similar questions, then explain what happens when a source changes or a user loses access after retrieval or report generation. #### What This Part Should Cover - Cache keys or eligibility checks that include identity, entitlements, source versions, and policy version. - Similarity thresholds and prevention of cross-user data leakage. - Reauthorization of cached evidence and saved reports at read time. - Invalidation, audit logs, redaction, and metrics for quality, cost, and security. ```hint Treat a cached answer as derived data Before reuse, prove that its evidence is still current and visible to the present caller; semantic similarity alone is not authorization. ``` ### What a Strong Answer Covers - End-to-end identity and authorization with no unauthorized model context. - High-recall retrieval, reranking, evidence lineage, and verifiable claim-level citations. - Bounded model planning with typed tools, evaluation, and safe failure behavior. - Permission-aware caching, revocation, report sharing, observability, and auditability. ### Follow-up Questions 1. How would you measure whether each generated claim is actually supported by its citation? 2. When can two semantically similar questions safely share retrieval or generation work? 3. What should happen to a saved report after one cited document becomes unauthorized? 4. How would you compare a fixed retrieval pipeline with a model-directed tool plan?