Design an A/B Test for Homepage Layout Impact

Q: Design an A/B Test for Homepage Layout Impact

This question evaluates a data scientist's skills in experimental design, causal inference, and product analytics by requiring specification of a primary success metric and guardrail metrics, execution of power analysis, planning for segmentation, and identification of validity threats such as sample-ratio mismatches, interference, novelty effects, and seasonality. It is commonly asked in Analytics & Experimentation interviews because it assesses practical application of randomization, estimands, and analysis plans while probing conceptual understanding of bias control and decision criteria; the domain tested is Analytics & Experimentation and the level combines practical application with conceptual understanding.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Design: New Homepage Layout → Purchase Rate

Context

You are designing an A/B test to evaluate whether a new homepage layout increases purchase rate for a high-traffic consumer website. The homepage is often the first touchpoint and changes may affect both conversion and user experience.

Task

Describe how you would design the experiment to measure the impact of the new layout on purchase rate. Address the following:

Population and exposure criteria (inclusions/exclusions)
Metrics
- Primary success metric
- Secondary/diagnostic metrics
- Guardrail metrics
Randomization strategy
- Unit of randomization, bucketing, consistency
Sample size and duration
- Power analysis and minimum detectable effect (MDE)
- Test duration and ramp-up plan
Bias control and validity threats
- Key risks and mitigations (e.g., SRM, interference, novelty, seasonality)
Analysis plan
- Estimand, statistical tests, confidence intervals
- Segmentation/heterogeneity analysis
- Multiple testing and decision criteria

Include explicit assumptions as needed. Mention success metric, guardrails, power analysis, segmentation, and validity threats.

Design an A/B Test for Homepage Layout Impact

Experiment Design: New Homepage Layout → Purchase Rate

Context

Task

Solution

Comments (0)