Compare Predictive Models from Two Vendors

# Compare Predictive Models from Two Vendors Two vendors provide models for the same prediction problem. How would you compare them and decide which model to use? The target, datasets, model types and business costs are not specified, so begin by identifying the information required for a fair comparison. Do not invent numerical performance results or assume a particular domain. Explain how you would evaluate both vendors on comparable data, choose metrics tied to the prediction objective, and distinguish a reliable improvement from variation caused by the sample or evaluation setup. Discuss how the recommendation would account for deployment constraints and outcomes on important subgroups, especially if the models trade off different kinds of errors. ### What a Strong Answer Covers - A shared target definition, prediction time and evaluation population. - Independent evaluation data, leakage checks and consistent feature availability. - Task-appropriate metrics, uncertainty on the comparison and subgroup performance. - Decision costs, operational requirements and a process for validating the selected model after deployment. ### Follow-up Questions - How would you compare the models if one ranks better but has worse probability calibration? - Why can vendor-reported scores on different datasets fail to establish which model is better?