Design A/B Test for New Amazon Recommendation Module
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Amazon wants to test a brand-new product recommendation on the home page.
##### Question
Design an A/B test to evaluate the new recommendation module. Define test and control, duration and required sample size. What primary metric would you track? How would you justify its business relevance? Define a p-value and explain how it is used to decide whether the experiment is successful. Describe potential sources of bias in this experiment and how you would guard against them. If you can only expose 5 % of users, how would you ensure adequate statistical power? Suppose the treatment lifts click-through-rate but reduces average order value; how would you decide whether to launch? Outline a causal inference approach (e.g., difference-in-differences or propensity matching) you could apply if randomization were impossible.
##### Hints
Cover hypothesis formulation, metric hierarchy, variance reduction, sequential testing and guardrail metrics.
Overview: Design A/B Test for New Amazon Recommendation Module evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
