This question evaluates a data scientist's skills in experimental design, causal inference, and product analytics by requiring specification of a primary success metric and guardrail metrics, execution of power analysis, planning for segmentation, and identification of validity threats such as sample-ratio mismatches, interference, novelty effects, and seasonality. It is commonly asked in Analytics & Experimentation interviews because it assesses practical application of randomization, estimands, and analysis plans while probing conceptual understanding of bias control and decision criteria; the domain tested is Analytics & Experimentation and the level combines practical application with conceptual understanding.
You are designing an A/B test to evaluate whether a new homepage layout increases purchase rate for a high-traffic consumer website. The homepage is often the first touchpoint and changes may affect both conversion and user experience.
Describe how you would design the experiment to measure the impact of the new layout on purchase rate. Address the following:
Include explicit assumptions as needed. Mention success metric, guardrails, power analysis, segmentation, and validity threats.
Login required