# Diagnose Strong Validation and Weak Production Performance You are given a training repository for a model whose validation metric looks strong while production performance is poor. Review the data and code paths, form ranked hypotheses, and explain how you would test and fix them. Include the role of data normalization and training-serving consistency. ### Constraints & Assumptions - The repository contains training, validation, and inference preprocessing code plus aggregate metrics. - Production labels may arrive late, and the live input distribution may differ from historical data. - The apparent gap can come from evaluation design, data leakage, preprocessing skew, distribution shift, or a product-metric mismatch. - A normalization change must be versioned with the model and safely deployable. ### Clarifying Questions to Ask - How are train and validation examples split, and can users, entities, or time windows overlap? - Does the offline metric represent the production decision and class distribution? - Are raw production features logged before and after preprocessing? - Where are normalization statistics fit, stored, and applied during inference? - When did the gap start, and does it vary by cohort, model version, or data source? ### Hints - Reproduce one example through training and serving transformations field by field. - Verify that every learned preprocessing statistic uses training data only. - Distinguish an offline-evaluation defect from genuine post-deployment drift. ### What a Strong Answer Covers - A ranked diagnostic plan using split integrity, leakage checks, cohort metrics, and training-serving parity. - Normalization fit only on training data, immutable saved statistics, feature ordering, missing-value behavior, and numeric stability. - Time-aware or entity-aware validation, production-representative metrics, calibration, and label-quality checks. - Distribution and concept drift, feedback loops, selection bias, and product-policy changes. - A controlled fix with backfill or shadow evaluation, canary rollout, monitoring, rollback, and reproducibility. - Evidence that distinguishes competing hypotheses rather than changing preprocessing by intuition. ### Follow-up Questions 1. How can normalization leak validation information even when model weights never see validation labels? 2. What evidence would distinguish training-serving skew from concept drift? 3. How would you handle a feature whose production distribution has a heavy new tail? 4. Which offline metric would you choose if the product acts only on the top one percent of scores?