Analyze an A/B test over last 7 days
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
Assume today is 2025-09-01. You ran a 50/50 A/B test on a checkout flow over the last 7 days (2025-08-26 to 2025-09-01). Daily exposures and purchases are below:
Date | A_exposed | A_purchases | B_exposed | B_purchases
-----------+-----------+-------------+-----------+------------
2025-08-26 | 28000 | 1350 | 27800 | 1420
2025-08-27 | 28500 | 1380 | 28100 | 1460
2025-08-28 | 28200 | 1390 | 27900 | 1450
2025-08-29 | 28400 | 1420 | 28000 | 1520
2025-08-30 | 28300 | 1370 | 27700 | 1480
2025-08-31 | 28100 | 1410 | 27900 | 1510
2025-09-01 | 30500 | 2580 | 30000 | 2560
Tasks:
1) Compute overall conversion rates for A and B, absolute/relative lift, and a two-proportion z-test p-value and 95% CI for the lift.
2) Check for sample ratio mismatch (SRM) daily and overall. If detected, propose root causes and mitigation.
3) Analyze heterogeneity across days; is it appropriate to pool? Justify with a fixed-effects vs random-effects framing.
4) Identify at least three pitfalls relevant to this window (e.g., seasonality/holiday effect on 2025-09-01, novelty effects, user overlap, peeking). Propose guardrails you would set before launching.
5) If your minimal detectable effect (MDE) was a +5% relative lift over baseline, assess achieved power approximately and whether you would roll out, iterate, or extend the test. State any assumptions.
Quick Answer: This question evaluates competency in A/B test analysis and experiment design, covering conversion-rate computation, statistical inference (two-proportion testing and confidence intervals), sample-ratio checks, heterogeneity assessment (fixed versus random effects), and power/MDE reasoning.
