A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)
Context: You will run an online experiment of a new checkout flow. The baseline conversion rate is 5%, and the product team expects a 3% relative lift (to 5.15%). You must select metrics, size the test, design randomization to avoid contamination, pre-register the analysis, and plan segmented readouts.
(a) Metrics
-
Choose one primary metric and at least two guardrail metrics; justify your choices.
(b) Sample Size
-
Compute the required sample size per group for 90% power at α = 0.05 (two-sided), given:
-
10% of traffic will be excluded as bot traffic post-exposure.
-
A 5% day-of-week (DOW) effect (seasonality) must be accounted for.
-
State all assumptions used.
(c) Randomization & Bucketing
-
Propose a scheme that prevents contamination across sessions and devices.
(d) Pre-registration
-
Define a pre-registration plan, including whether you use a fixed-horizon or sequential design, and how peeking will be handled (e.g., group sequential or alpha spending).
(e) Segmentation & Multiplicity
-
Describe how you will segment results (e.g., new vs returning) while controlling for multiplicity.