Design a Production Retrieval-Augmented Generation Pipeline

## Scenario Design a production retrieval-augmented generation pipeline over documents stored in cloud object storage. Users ask questions through an API, the system retrieves relevant passages from a vector-capable index, and a language model produces an answer grounded in those passages. Cover ingestion, retrieval strategy, vector storage, model orchestration, agent boundaries, reliability, and evaluation. Do not assume that an agent should autonomously call tools unless the use case requires it. ### Constraints & Assumptions - Documents are versioned and may be corrected or deleted. - Users have different document permissions. - Parsing and embedding are asynchronous; queries require a stated freshness target. - Duplicate events, partial ingestion failures, and model or embedding-provider rate limits are expected. - The design may use managed cloud services, but explain the required capability rather than relying on a product name as the architecture. ### Clarifying Questions to Ask - What document types, corpus size, update rate, languages, and permission model are expected? - Must every answer cite source passages and refuse when evidence is weak? - What latency, availability, retention, and cost targets matter? - Is the model allowed to use external tools, or is retrieval the only augmentation? - How will answer quality be evaluated before and after launch? ```hint Give every derived artifact a source version Chunks, embeddings, index entries, and cached answers should be traceable to the exact document version that produced them. ``` ### What a Strong Answer Covers - Durable ingestion events, idempotent parsing, chunking choices, metadata, embedding, indexing, and deletion propagation. - Hybrid candidate retrieval, permission filtering, reranking, context budgeting, and citation binding. - A source of truth distinct from replaceable vector and keyword indexes. - Backpressure, retries, dead-letter handling, checkpointing, and reconciliation for partial failures. - Clear conditions under which a workflow remains a deterministic pipeline or becomes a tool-using agent. - Offline retrieval and answer-quality evaluation, adversarial permission tests, online feedback, and observability. - Cost and latency controls such as batching, caching, tiered models, and bounded context. ### Follow-up Questions 1. How do you remove every embedding and cache entry after a document is revoked? 2. When does keyword retrieval outperform vectors, and how would you combine them? 3. How would you diagnose a grounded but irrelevant answer? 4. What state and permission checks must surround an agent tool call? 5. How would you migrate to a new embedding model without mixing incompatible vectors?