Design an A/B test with guardrails

Q: Design an A/B test with guardrails

This question evaluates a data scientist's skills in experimental design, hypothesis testing, statistical power and sample-size calculation, randomization and contamination control, pre-registration planning, and multiplicity adjustment for A/B testing.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)

Context: You will run an online experiment of a new checkout flow. The baseline conversion rate is 5%, and the product team expects a 3% relative lift (to 5.15%). You must select metrics, size the test, design randomization to avoid contamination, pre-register the analysis, and plan segmented readouts.

(a) Metrics

Choose one primary metric and at least two guardrail metrics; justify your choices.

(b) Sample Size

Compute the required sample size per group for 90% power at α = 0.05 (two-sided), given:
- 10% of traffic will be excluded as bot traffic post-exposure.
- A 5% day-of-week (DOW) effect (seasonality) must be accounted for.
State all assumptions used.

(c) Randomization & Bucketing

Propose a scheme that prevents contamination across sessions and devices.

(d) Pre-registration

Define a pre-registration plan, including whether you use a fixed-horizon or sequential design, and how peeking will be handled (e.g., group sequential or alpha spending).

(e) Segmentation & Multiplicity

Describe how you will segment results (e.g., new vs returning) while controlling for multiplicity.

Design an A/B test with guardrails

A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)

Solution

Comments (0)

Design an A/B test with guardrails

Overview

A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)

Solution

Comments (0)