Amazon wants to test a brand-new product recommendation on the home page.

Design an A/B test to evaluate the new recommendation module.  Define test and control, duration and required sample size. What primary metric would you track?  How would you justify its business relevance? Define a p-value and explain how it is used to decide whether the experiment is successful. Describe potential sources of bias in this experiment and how you would guard against them. If you can only expose 5 % of users, how would you ensure adequate statistical power? Suppose the treatment lifts click-through-rate but reduces average order value; how would you decide whether to launch? Outline a causal inference approach (e.g., difference-in-differences or propensity matching) you could apply if randomization were impossible.

Cover hypothesis formulation, metric hierarchy, variance reduction, sequential testing and guardrail metrics.

Amazon Data Scientist interview question: Design A/B Test for New Amazon Recommendation Module. {"blocks": [{"key": "4742b255", "text": "Scenario", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {...