Choosing Offline and Online Metrics to Evaluate a Generative AI Feature's Output

You are the product manager for an AI product whose core output is text generated by a large language model. How would you judge whether that generated text is good? Name the evaluation metrics you would use, explain how each one is measured, and describe how you would combine them to decide whether a model or prompt change is ready to ship and whether quality is holding up after launch. ```hint Quality depends on the task "Good output" means different things for a summarizer, a coding assistant and a support bot. Pick one concrete product first, list what its users would call a bad answer, and derive the metrics from those failure modes. ``` ### Clarifying Questions - Which product is this: an open-ended chat assistant, a summarizer, a coding assistant, or an assistant that answers from a fixed set of documents? - Are we evaluating before launch to choose a model and prompt, monitoring quality after launch, or both? - Do we have reference answers or trained human raters available, and how much labeling budget is there? - Which failures are most costly for this product: factual errors, unsafe content, or answers that are simply unhelpful? ### What a Strong Answer Covers - Breaking generation quality into distinct dimensions (correctness, grounding, relevance, instruction following, safety, tone) tied to the chosen product - Offline methods and their limits: reference-overlap scores, human rubric and side-by-side ratings, model-graded evaluation and task-specific checks - Online user signals that reveal quality in production, and how to read them through an A/B test - A north-star metric for the product plus guardrails for safety, latency and cost - How the metric set stays trustworthy: a representative evaluation set, rater agreement, and calibration of automated judges against human labels ### Follow-up Questions - Your automated judge scores a new model higher, but users regenerate its answers more often. What do you investigate? - How would you measure hallucination for an assistant that answers from a company's own documents? - How would you keep the evaluation set from going stale as user behavior and the product change?