Design an A/B Test for Homepage Layout Impact
Overview
This question evaluates a data scientist's skills in experimental design, causal inference, and product analytics by requiring specification of a primary success metric and guardrail metrics, execution of power analysis, planning for segmentation, and identification of validity threats such as sample-ratio mismatches, interference, novelty effects, and seasonality. It is commonly asked in Analytics & Experimentation interviews because it assesses practical application of randomization, estimands, and analysis plans while probing conceptual understanding of bias control and decision criteria; the domain tested is Analytics & Experimentation and the level combines practical application with conceptual understanding.
Experiment Design: New Homepage Layout → Purchase Rate
Context
You are designing an A/B test to evaluate whether a new homepage layout increases purchase rate for a high-traffic consumer website. The homepage is often the first touchpoint and changes may affect both conversion and user experience.
Task
Describe how you would design the experiment to measure the impact of the new layout on purchase rate. Address the following:
- Population and exposure criteria (inclusions/exclusions)
-
Metrics
- Primary success metric
- Secondary/diagnostic metrics
- Guardrail metrics
-
Randomization strategy
- Unit of randomization, bucketing, consistency
-
Sample size and duration
- Power analysis and minimum detectable effect (MDE)
- Test duration and ramp-up plan
-
Bias control and validity threats
- Key risks and mitigations (e.g., SRM, interference, novelty, seasonality)
-
Analysis plan
- Estimand, statistical tests, confidence intervals
- Segmentation/heterogeneity analysis
- Multiple testing and decision criteria
Include explicit assumptions as needed. Mention success metric, guardrails, power analysis, segmentation, and validity threats.
Solution
Login required