Walk through the loss, offline and online metrics behind an ML model you worked on

Pick a model from a product you worked on. Which metrics did you use to train it, and which did you use to evaluate it? ```hint Count the layers Ask what the optimizer actually minimized, what told you a new version beat the old one, and what told you users were better off. Check whether those were the same number for your model, and be ready to explain why not. ``` ### Clarifying Questions - Should the example be a classic predictive model (classification, ranking, forecasting) or a generative model such as an LLM feature, where there is no single correct output? - Do you want builder-level depth (loss functions, validation splits, thresholds) or the product view of how the metrics drove the ship decision? - Does a model I evaluated and launched but did not train myself, such as a hosted foundation model behind a feature, count as an example? - Should the answer cover how the model was monitored after launch, or stop at the ship decision? ### What a Strong Answer Covers - A concrete model and the product decision it served, so each metric has a stated reason to exist - A clear split between the training objective, the offline evaluation metrics and the online product metrics, and how they connect - Why each metric suits the task: class imbalance, the different costs of false positives and false negatives, ranking versus classification, open-ended generated output - How the evaluation data was built and kept honest: held-out or golden sets, leakage, and results by segment - Guardrails such as latency, serving cost and safety, and what happened when offline and online results disagreed ### Follow-up Questions - Your offline metric improved but the online test showed no lift. What are the likely causes, and what would you do next? - How did you decide the threshold at which the model was good enough to ship? - How would you build an evaluation set for this model if no labeled data existed at the start?