Analyze an A/B test over last 7 days
Overview
This question evaluates competency in A/B test analysis and experiment design, covering conversion-rate computation, statistical inference (two-proportion testing and confidence intervals), sample-ratio checks, heterogeneity assessment (fixed versus random effects), and power/MDE reasoning.
Amazon
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
A/B Test Readout and Decision (2025-08-26 to 2025-09-01)
Context
A 50/50 A/B experiment on the checkout flow ran for 7 days, from 2025-08-26 through 2025-09-01 (today). Below are daily exposures and purchases for variants A and B. Treat exposures as the unit of randomization/analysis and purchases as binary conversions within the analysis window.
| Date | A_exposed | A_purchases | B_exposed | B_purchases |
|---|---|---|---|---|
| 2025-08-26 | 28,000 | 1,350 | 27,800 | 1,420 |
| 2025-08-27 | 28,500 | 1,380 | 28,100 | 1,460 |
| 2025-08-28 | 28,200 | 1,390 | 27,900 | 1,450 |
| 2025-08-29 | 28,400 | 1,420 | 28,000 | 1,520 |
| 2025-08-30 | 28,300 | 1,370 | 27,700 | 1,480 |
| 2025-08-31 | 28,100 | 1,410 | 27,900 | 1,510 |
| 2025-09-01 | 30,500 | 2,580 | 30,000 | 2,560 |
Tasks
- Compute overall conversion rates (CR) for A and B, absolute/relative lift, and a two-proportion z-test p-value and 95% CI for the lift.
- Check for sample ratio mismatch (SRM) daily and overall. If detected, propose root causes and mitigation.
- Analyze heterogeneity across days; is it appropriate to pool? Justify with a fixed-effects vs random-effects framing.
- Identify at least three pitfalls relevant to this window (e.g., seasonality/holiday effect on 2025-09-01, novelty effects, user overlap, peeking). Propose guardrails before launch.
- If the minimal detectable effect (MDE) was +5% relative over baseline, assess achieved power approximately and recommend whether to roll out, iterate, or extend. State assumptions.
Solution
Login required
Show