Design an A/B test with guardrails

Quick Overview

This question evaluates a data scientist's skills in experimental design, hypothesis testing, statistical power and sample-size calculation, randomization and contamination control, pre-registration planning, and multiplicity adjustment for A/B testing.

Design an A/B test with guardrails

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Your team will test a new checkout flow expected to improve conversion by 3% relative (baseline 5%). (a) Choose a primary metric and at least two guardrail metrics; justify. (b) Compute the required sample size per group for 90% power at α=0.05, accounting for a 10% bot-traffic exclusion and a 5% day-of-week effect; specify all assumptions. (c) Propose a randomization and bucketing scheme that prevents contamination across sessions and devices. (d) Define a pre-registration plan, including a fixed-horizon vs sequential design and how you will handle peeking (e.g., group sequential or alpha spending). (e) Describe how you will segment results (e.g., new vs returning) while controlling multiplicity.

Quick Answer: This question evaluates a data scientist's skills in experimental design, hypothesis testing, statistical power and sample-size calculation, randomization and contamination control, pre-registration planning, and multiplicity adjustment for A/B testing.

|Home/Analytics & Experimentation
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
4
0

A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)

Context: You will run an online experiment of a new checkout flow. The baseline conversion rate is 5%, and the product team expects a 3% relative lift (to 5.15%). You must select metrics, size the test, design randomization to avoid contamination, pre-register the analysis, and plan segmented readouts.

(a) Metrics

  • Choose one primary metric and at least two guardrail metrics; justify your choices.

(b) Sample Size

  • Compute the required sample size per group for 90% power at α = 0.05 (two-sided), given:
    • 10% of traffic will be excluded as bot traffic post-exposure.
    • A 5% day-of-week (DOW) effect (seasonality) must be accounted for.
  • State all assumptions used.

(c) Randomization & Bucketing

  • Propose a scheme that prevents contamination across sessions and devices.

(d) Pre-registration

  • Define a pre-registration plan, including whether you use a fixed-horizon or sequential design, and how peeking will be handled (e.g., group sequential or alpha spending).

(e) Segmentation & Multiplicity

  • Describe how you will segment results (e.g., new vs returning) while controlling for multiplicity.
Loading comments...