Compare Predictive Models from Two Vendors
Company: Google
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Onsite
# Compare Predictive Models from Two Vendors
Two vendors provide models for the same prediction problem. How would you compare them and decide which model to use? The target, datasets, model types and business costs are not specified, so begin by identifying the information required for a fair comparison. Do not invent numerical performance results or assume a particular domain.
Explain how you would evaluate both vendors on comparable data, choose metrics tied to the prediction objective, and distinguish a reliable improvement from variation caused by the sample or evaluation setup. Discuss how the recommendation would account for deployment constraints and outcomes on important subgroups, especially if the models trade off different kinds of errors.
### What a Strong Answer Covers
- A shared target definition, prediction time and evaluation population.
- Independent evaluation data, leakage checks and consistent feature availability.
- Task-appropriate metrics, uncertainty on the comparison and subgroup performance.
- Decision costs, operational requirements and a process for validating the selected model after deployment.
### Follow-up Questions
- How would you compare the models if one ranks better but has worse probability calibration?
- Why can vendor-reported scores on different datasets fail to establish which model is better?
Overview: Design a fair comparison of two vendors’ predictive models using common data, relevant metrics, uncertainty and deployment constraints.
Read the full Google Data Scientist interview experience this question came from
Community answers
Answer by shishirpuri881
so if we have two models, then the major evaluation we need to carry out before deciding which one to go up with would definitely be the mathematical accuracy and business problem solving approach. We can rely on mathematical indices like accuracy, precision, recall in case of classification and in classification also we would give priority to recall or preceision as per the problem type, and in case of regression then RMSE, R2 score and like that. but these all are mathematical metrices, so for real world business approaches we need to have look over the business overview also, we need to look over which model works or fits as per the real world natural tendency of work, and also there comes computational cost, latency, and performanc also, so considering all these things we would decide which model to go up with