Evaluate a Retrieval-Augmented Generation System

Design an evaluation framework for a retrieval-augmented generation system that answers questions from an authorized document corpus. Evaluate retrieval, answer generation, citation support, abstention, latency, and production impact. Explain how you would build the test set and prevent an answer-quality metric from hiding retrieval failures. ### Constraints & Assumptions - Some questions are answerable, some require multiple documents, and some are unanswerable from the corpus. - Documents change over time and access controls must be preserved. - Automated judges may be used only with calibration against human review. ### Clarifying Questions to Ask - What user task and harm level define a correct answer? - Must every factual claim cite an exact supporting passage? - What retrieval cutoff and latency budget are available? - How often do documents and permissions change? ### Part 1 — Retrieval evaluation Measure whether the system finds the evidence needed to answer. #### What This Part Should Cover - Query-document or query-passage relevance labels. - Recall, MRR, and nDCG at explicit cutoffs. - Multi-hop coverage, hard negatives, freshness, and permission-filter tests. ### Part 2 — Generation and grounding Measure answer usefulness without rewarding unsupported fluency. #### What This Part Should Cover - Correctness, completeness, claim-level support, citation precision and recall, and abstention. - Evaluation both with oracle context and with retrieved context to isolate failures. - Human rubric and calibrated automated-judge limitations. ### Part 3 — Online system quality Define rollout and monitoring metrics. #### What This Part Should Cover - Task success, escalation or correction, user trust signals, latency, cost, and safety. - A/B or phased rollout with logging that respects privacy and permissions. - Drift by query type, corpus version, and retrieval route. ```hint Use an evaluation matrix Score the generator with gold evidence and with actual retrieved evidence. The gap separates generation limitations from retrieval misses. ``` ### What a Strong Answer Covers - Component-level and end-to-end metrics tied to a representative test set. - Unanswerable questions and calibrated refusal behavior. - Claim-level grounding rather than string overlap alone. - Access-control, freshness, latency, and cost as first-class quality dimensions. ### Follow-up Questions - How would you evaluate a question whose answer changes with document version? - What makes a useful hard negative for retrieval? - How can citation correctness be measured when several passages support the same claim? - When should a RAG system route to a human instead of answering?