Debug online worse than offline model performance
Quick Overview
This question evaluates the ability to diagnose discrepancies between offline and online model performance by reasoning about data distributions, feature computation, serving infrastructure, evaluation methodology, and feedback loops, testing skills in debugging model deployment and operational analytics.
Amazon
Jan 6, 2026, 12:00 AM
Machine Learning Engineer
Onsite
ML System Design
3
0
Loading...
Production ML: online performance worse than offline
You launch an ML model. Offline evaluation (validation/test) looked good, but after deployment the online metrics are significantly worse.
Question
What step-by-step troubleshooting order would you follow to identify and fix the issue? Include what you would check in data, features, serving, evaluation, and feedback loops.
Solution
Show