{"blocks": [{"key": "cfedbed9", "text": "Scenario", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}, {"key": "e723678e", "text": "Meta’s ads platform team claims a newly-built ML ranking model will outperform the current recommendation system.", "type": "unstyled", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}, {"key": "070bc5ef", "text": "Question", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}, {"key": "abd27c5b", "text": "How would you design an experiment to evaluate the new recommendation model? Specify control / treatment assignment, duration and success metrics. Which primary, guard-rail and long-term metrics would you choose and why? If an A/B test shows a 5 % lift in overall CTR, how would you quantify the business gain? The test reports a 100 % CTR increase for males aged 18-55 in India. How do you interpret this result? If the experiment shows +5 % CTR and +5 % revenue with no negative guard-rail impact, would you roll out the model? Explain.", "type": "unstyled", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}, {"key": "4934f478", "text": "Hints", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}, {"key": "362d0edd", "text": "Discuss sample size, statistical significance, heterogeneity, north-star vs. guard-rail metrics and risk mitigation.", "type": "unstyled", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {}}], "entityMap": {}}