Design an Evaluation and Improvement Loop for a Production LLM Application

You are in the ML system design round of a machine learning loop for LLM products. The problem is about evaluating and improving an LLM system: design what a team would use to measure how well an LLM-powered application performs, find where it fails, and turn those findings into improvements that ship safely. Expect the interviewer to ask you to draw the architecture while you explain it, so be ready to sketch the components and data flows rather than only describe them. The exact application was not reported. Unless the interviewer specifies otherwise, assume a user-facing assistant that answers questions by retrieving passages from a document collection and generating a response with an LLM, and a team that regularly changes its prompts, model and retrieval settings. ### Constraints and Clarifications - Treat "improving" as covering changes to prompts, the base model, retrieval and chunking settings, and guardrails; assume each change has to be shown to help before it reaches all users. - The design has to work before release (offline evaluation) and after it (production monitoring). - Assume human raters are available but slow and limited, so they cannot score every response. - Assume production conversations can contain personal data. ### Clarifying Questions - What does the application do, who uses it, and what does a good response look like to them: correct, grounded in the documents, concise, safe? - Which failures hurt most today: invented facts, irrelevant retrieval, unnecessary refusals, unsafe content, latency or cost? - Do reference answers or labeled examples already exist, or does the evaluation data have to be built from scratch? - How often do prompt or model changes ship, and who has to approve them? - What feedback does the product already capture, such as ratings, edits, retries or escalations to a human? ### Part 1 — Define quality and build offline evaluation Decide what to measure and how to measure it before a change reaches users. Describe the evaluation datasets, the metrics, who or what scores responses, and how a proposed change is compared with the current version. ```hint Agree on quality first Break response quality into a few dimensions that can each be scored, then ask which of them need a reference answer and which can be judged from the question and the retrieved passages alone. ``` #### What This Part Should Cover - A small set of quality dimensions tied to the product, plus latency and cost - How evaluation data is sourced, labeled, versioned and kept current - Who or what scores responses, and how the scorers are validated - A release decision that rests on enough data to trust the measured difference ### Part 2 — Evaluate and monitor in production Describe how you would measure quality on live traffic, detect regressions and drift, and test changes on real users. ```hint Live traffic has no labels Think about which signals arrive for free with each interaction, which need sampling and scoring, and how you would tell a real regression from noise. ``` #### What This Part Should Cover - What is recorded per request so results can be attributed to specific versions - Which quality signals are collected on live traffic, and at what cost - How changes are tested on real users and what protects those users - How regressions and drift are detected and surfaced ### Part 3 — Close the improvement loop and draw the system Explain how failures found offline or online become fixes, and how those fixes are validated and rolled out. Then draw the end-to-end architecture that ties Parts 1 to 3 together. ```hint Every failure should become a test Follow one bad response from the moment it is detected to the change that fixes it, and note where it should be kept so the same bug cannot quietly return. ``` #### What This Part Should Cover - How failures are grouped and attributed to the component that caused them - How the fix is chosen for each kind of failure - How fixed failures are kept from returning, and how fixes ship safely - A clear diagram showing data flowing between the serving, evaluation and feedback components ### What a Strong Answer Covers - Clarifies the application and the cost of each failure type before proposing components - Treats evaluation as measurement: versioned datasets, validated judges, and confidence in comparisons - Separates retrieval quality from generation quality so failures can be attributed - Balances evaluation cost against coverage - Handles the privacy of logged conversations and the risk of over-optimizing a proxy metric - Communicates through a structured, drawn design rather than a stream of ideas ### Follow-up Questions - How do you confirm that an LLM judge agrees with human raters, and what do you do when the judge model itself is upgraded? - Offline scores improved, but the online experiment shows no change in user outcomes. How do you investigate? - How would you extend the evaluation to multi-turn conversations, or to an agent that calls tools? - Scoring every production response with a large judge model is too expensive. How do you decide what to score?