Measuring the Health of a Production ML System, From Data Quality to Business Impact

You are the product manager for a product that relies on a machine learning pipeline. Which metrics would you track to judge how well that pipeline is performing? ```hint Follow a prediction end to end A pipeline is more than the model. Trace one prediction from the raw data that feeds it to the moment it affects a user, and ask what could quietly go wrong at each step and which number would reveal it first. ``` ### Clarifying Questions - What does the pipeline predict, and is it a classification, ranking or regression problem? That decides which model-quality metrics make sense. - Does it serve predictions in real time inside the product, or run as a batch job that writes scores for later use? - How soon after a prediction does the true outcome become known: within seconds (a click), weeks later (a purchase or a churned account), or only through manual review? - Who will act on these metrics: the engineers on call for the pipeline, or business owners deciding whether the model is worth keeping? ### What a Strong Answer Covers - Coverage of the whole pipeline: input data quality, model quality before and after release, serving and training health, and business impact - Model-quality metrics matched to the problem type and to the cost of each kind of error, rather than a default accuracy figure - How to detect degradation when true outcomes arrive late or never, including drift and proxy signals - A clear hierarchy (primary success metric, release gates, guardrails) with each metric tied to an owner, a threshold or a decision - Trade-offs among model quality, latency and cost, and how the metric set exposes them ### Follow-up Questions - A new model version beats the current one on offline precision and recall, yet the business metric does not move after launch. How would you investigate? - True outcomes for this model arrive only weeks after each prediction. How would you notice a problem before then? - Leadership wants a single number on a dashboard that says the pipeline is healthy. What would you choose, and what would you warn them it hides?