Walk through an A/B test end-to-end
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
Walk through how you would design, run, and analyze an **A/B test** for a product change.
Your answer should include:
- Hypothesis framing and choosing **primary**, **diagnostic**, and **guardrail** metrics.
- Experiment design: unit of randomization, population, exposure definition, duration, and handling novelty/seasonality.
- How you determine **sample size / MDE / power**.
- Data quality checks (e.g., SRM), logging issues, and how you validate randomization.
- Statistical analysis approach (confidence intervals, p-values, multiple testing, sequential peeking).
- How you interpret results and make a launch decision, including practical vs statistical significance.
- Common pitfalls (e.g., interference/network effects, noncompliance, missing data).
Quick Answer: This question evaluates a data scientist's competencies in experimental design, statistical inference, causal reasoning, metric selection, and data quality assurance for A/B testing.
