Validate and monitor ranking model end-to-end
Company: Expedia
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
For the same Expedia hotel-ranking model: a) Propose an offline evaluation plan that prevents leakage (time-based splits, user-level grouping), handles position bias (propensity-weighted/IPS metrics or counterfactual LTR), and reports probability calibration of conversion predictions (e.g., reliability curves, ECE). b) Specify which ranking metrics you’ll report (e.g., NDCG@10 with revenue weights, ERR), including the formulas and why they reflect client value. c) Outline diagnostics to identify key drivers without leaking target information (e.g., SHAP with proper background, permutation checks, stability across folds). d) Define rollout/monitoring: shadow mode, canary, guardrail alarms, drift detection (data and concept), late-arriving data handling, and an automatic rollback policy with thresholds. e) Describe how you’d verify that optimizing a surrogate objective still improves the client KPI, and what you’d do if the offline–online relationship breaks.
Overview: This question evaluates a data scientist's mastery of learning-to-rank concepts, offline evaluation and metric design, position-bias correction, diagnostic analysis, deployment safeguards, and alignment of surrogate objectives with business KPIs for hotel search ranking.
Read the full Expedia Data Scientist interview experience this question came from