Design an Experiment to Evaluate New ML Model

##### Scenario Ads platform wants to validate engineers’ claim that a new ML ranking model outperforms the existing recommender. ##### Question How would you design an experiment to evaluate the new ML recommendation system? Which primary and guard-rail metrics would you monitor and why? An A/B test shows a 5% lift in CTR—how do you judge practical significance? CTR doubles for Indian males aged 18-55—what might this indicate and what next steps would you take? If the test yields +5% CTR and +5% revenue, would you roll the model out globally? Explain your decision process. ##### Hints Discuss randomization, sample size, heterogeneous effects, business trade-offs, and ethical checks.