Evaluating an LLM in Production: Offline Test Sets, Graders and Live Product Metrics

How would you evaluate how well an LLM you are responsible for is performing? Walk through the methods and the metrics you would rely on. ```hint Start from what the model is for Ask what the model is used for before you name a metric. A score that suits translation or next-word prediction may say little about whether a support assistant answers correctly and safely. ``` ### Clarifying Questions - What is the model used for: open-ended chat, summarization, pulling fields out of documents, or an agent that takes actions? - Are we training or fine-tuning our own model, or calling a third-party model through an API and controlling only the prompt, retrieval and model choice? - Is there a labeled set of inputs with known-good outputs, or do we have to build one? - Is the evaluation for choosing a model before launch, or for monitoring one that is already live? ### What a Strong Answer Covers - Tying the evaluation to the use case and to what a good output means for users, rather than listing every known metric. - The difference between model-level measures (perplexity, cross-entropy) and task-level measures (exact match, overlap scores, human or model-graded quality), and the limits of each. - How the offline test set is built and graded, including human review and calibration of any LLM-as-judge grader. - Production metrics tied to user and business outcomes, with guardrails for hallucination, safety, latency and cost. - Evaluation as a continuous loop: regression gates on prompt or model changes, drift monitoring, and feeding failures back into the test set. ### Follow-up Questions - After a model upgrade your LLM-graded score goes up, but user complaints also rise. How do you investigate? - How would you measure hallucination for a feature that answers questions from company documents? - A cheaper model scores slightly lower on your offline evaluation. How do you decide whether to switch?