Evaluating a Production LLM: Quality Dimensions, Offline Evals and Live Monitoring
Company: Google
Role: Product Manager
Category: Machine Learning
Difficulty: medium
Interview Round: Other
In a product manager interview, you are asked to walk through how you have evaluated the quality and performance of a large language model (LLM) running in a production product. Use a product you worked on: explain how you defined good output, how you measured it before and after release, and what you changed because of what you found.
```hint Test set versus live traffic
Some failures show up in a curated test set; others appear only with real users, real data and changes to the model or content over time. Organize your walkthrough around how each kind of failure was caught and what it made you change.
```
### Clarifying Questions
- Should I go deep on one LLM feature, or summarize evaluation practices across several?
- How technical should I be: the evaluation design and metrics only, or also the tooling and pipelines behind them?
- Does "performance" mean answer quality, serving characteristics such as latency and cost, or both?
- If my hands-on production experience is limited, would you prefer I describe what I did and how I would extend it, or walk through a full process for a concrete use case?
### What a Strong Answer Covers
- A definition of quality specific to the use case, broken into dimensions such as task success, factual grounding, safety and format, alongside latency, cost and reliability
- Offline evaluation: a representative test set, rubric-based human review, automated graders calibrated against people, and regression gates for prompt or model changes
- Online evaluation: staged rollout or experiments tied to user outcomes, implicit and explicit feedback, and sampled review of live traffic
- Continuous monitoring for drift and regressions, with a loop that turns production failures into fixes and new test cases
- The candidate's own role and decisions, including a trade-off they made and results they can quantify
### Follow-up Questions
- Your automated grader and your human reviewers disagree on a large share of cases. What do you do?
- A new model version scores better offline, but users escalate to a human more often after rollout. How do you find out why?
- How did you balance logging enough production data to judge quality against user privacy?
Overview: A product manager interview question asking candidates to walk through how they evaluated a large language model running in production. It tests defining quality for a use case, offline test sets and calibrated graders, staged rollouts and experiments, monitoring for drift, and the trade-offs the candidate made.