Validate and monitor ranking model end-to-end

For the same Expedia hotel-ranking model: a) Propose an offline evaluation plan that prevents leakage (time-based splits, user-level grouping), handles position bias (propensity-weighted/IPS metrics or counterfactual LTR), and reports probability calibration of conversion predictions (e.g., reliability curves, ECE). b) Specify which ranking metrics you’ll report (e.g., NDCG@10 with revenue weights, ERR), including the formulas and why they reflect client value. c) Outline diagnostics to identify key drivers without leaking target information (e.g., SHAP with proper background, permutation checks, stability across folds). d) Define rollout/monitoring: shadow mode, canary, guardrail alarms, drift detection (data and concept), late-arriving data handling, and an automatic rollback policy with thresholds. e) Describe how you’d verify that optimizing a surrogate objective still improves the client KPI, and what you’d do if the offline–online relationship breaks.