Design and Evaluate a Retrieval-Augmented Generation Pipeline

Design a retrieval-augmented generation pipeline over a document collection. Explain document chunking, retrieval, context assembly, and how you would evaluate retrieval recall separately from answer quality. ### Constraints & Assumptions - The system answers questions using a versioned document collection and should identify the evidence supporting its answers. - The source calls out chunking and evaluation/recall metrics but does not provide a document domain, model, scale, or quality target. State any baseline choices explicitly. - Treat documents as untrusted data, and preserve any access restrictions when selecting context. - Explain how the system responds when sufficient evidence is absent; a fluent answer alone is not a success criterion. ### Clarifying Questions to Ask - What kinds of documents and questions must be supported, and how often does the corpus change? - Is relevance defined at the document, passage, or supporting-evidence level? - Which users may retrieve which documents, and can those permissions change? - What latency and context-size limits constrain the amount of retrieved material? ```hint Measure the pipeline in stages A correct answer can fail because the evidence was never indexed, was not retrieved, was removed during context assembly, or was ignored during generation. Design evaluation that separates those cases. ``` ### What a Strong Answer Covers - Parsing, chunk boundaries, overlap, source offsets, versioning, and indexing. - Retrieval and optional reranking with permission filtering and context deduplication. - A definition of recall at a fixed retrieval depth and its dependence on relevance labels. - Answer correctness, grounding, citation accuracy, and abstention evaluation. - A controlled method for comparing chunking strategies without leaking evaluation questions into tuning. ### Follow-up Questions - Why might smaller chunks improve one retrieval metric but harm the final answer? - How would stale or deleted documents be removed from both retrieval and cached responses?