Measuring answer quality, latency and cost for a RAG-based LLM product

Think of an LLM or retrieval-augmented generation (RAG) system you have helped build. How would you measure whether it is efficient? ```hint Unpack the word "efficient" Decide what efficient should mean for this system before naming any metric: good answers, cheap and fast answers, or value delivered for what it costs. A RAG pipeline has more than one stage that can fail, so think about how you would locate the problem when the final answer is poor. ``` ### Clarifying Questions - Should "efficient" cover answer quality, serving cost and latency, the business value delivered for the spend, or all three? - Who uses the system: employees searching internal documents, customers in a support channel, or developers calling it through an API? - Is there already a labeled set of questions with reference answers and source passages, or would it have to be built? - Is the model called through a third-party API billed per token, or self-hosted on the team's own GPUs? ### What a Strong Answer Covers - A definition of efficiency for this system that separates answer quality from cost and speed and ties both to the project's goal - Separate offline evaluation of retrieval and of generation, including how the evaluation set is built and kept trustworthy - Cost and performance measures such as latency, tokens and cost per useful answer, and the levers that move them - Online user and business outcomes, with guardrails for hallucination, safety and regressions - How the metrics drive decisions: baselines, quality-versus-cost trade-offs and ongoing monitoring ### Follow-up Questions - A smaller model halves cost per query, but your offline faithfulness score drops a few points. How do you decide whether to switch? - Retrieval recall looks strong offline, yet users often rephrase and ask again. What would you investigate? - How would you keep the evaluation set from going stale as the documents and the questions users ask change?