Retrieval-Augmented Generation Quality: Offline and Online Evaluation Plan

Your team has built a retrieval-augmented generation (RAG) pipeline: it finds documents relevant to a user's question and has a large language model write an answer from them. How would you evaluate it? ```hint Where did it go wrong? A bad answer can come from either half of the pipeline. Think about what you would need to record and label to tell which half failed on a given question. ``` ### Clarifying Questions - What does the pipeline answer, and for whom: customers asking about the company's security products, or internal analysts working through alerts with runbooks and threat intelligence? - Is there already a set of questions with known correct answers and known relevant documents, or does it need to be built? - Is the evaluation for a launch decision, or for comparing changes such as a new chunking scheme, embedding model or language model? - Do users have different access rights to the documents the pipeline searches? ### What a Strong Answer Covers - Separate evaluation of retrieval, of generation and of end-to-end answer quality, with named metrics for each. - How a representative evaluation set is built, including unanswerable, multi-step, outdated-version and adversarial questions. - How answers are graded at scale: expert labels, model-based judges checked against humans, and automated checks. - Online evaluation after launch: user outcome signals, experiments, and guardrails for latency, cost, freshness and access control. - Launch bars and a regression process tied to the cost of a wrong answer in this use case. ### Follow-up Questions - Groundedness scores are high, yet users still report wrong answers. Where do you look first? - The document collection changes every day. How do you keep both the evaluation set and the scores trustworthy? - How would you show that the pipeline never surfaces content from a document the asking user is not allowed to see?