PracHub
QuestionsPremiumLearningGuidesInterview PrepNEWCoaches
|Home/Analytics & Experimentation/Other

Design an A/B test with guardrails

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's skills in experimental design, hypothesis testing, statistical power and sample-size calculation, randomization and contamination control, pre-registration planning, and multiplicity adjustment for A/B testing.

  • hard
  • Other
  • Analytics & Experimentation
  • Data Scientist

Design an A/B test with guardrails

Company: Other

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Your team will test a new checkout flow expected to improve conversion by 3% relative (baseline 5%). (a) Choose a primary metric and at least two guardrail metrics; justify. (b) Compute the required sample size per group for 90% power at α=0.05, accounting for a 10% bot-traffic exclusion and a 5% day-of-week effect; specify all assumptions. (c) Propose a randomization and bucketing scheme that prevents contamination across sessions and devices. (d) Define a pre-registration plan, including a fixed-horizon vs sequential design and how you will handle peeking (e.g., group sequential or alpha spending). (e) Describe how you will segment results (e.g., new vs returning) while controlling multiplicity.

Quick Answer: This question evaluates a data scientist's skills in experimental design, hypothesis testing, statistical power and sample-size calculation, randomization and contamination control, pre-registration planning, and multiplicity adjustment for A/B testing.

Related Interview Questions

  • Design metrics resilient to data quality - Other (Medium)
  • Design and power an A/B on question mix - Other (medium)
  • Find and fix metric drops systematically - Other (medium)
  • Separate demand from supply for jeans - Other (medium)
  • Design an A/B test for a Celebrate reaction - Other (hard)
Other logo
Other
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
2
0

A/B Test Plan for a New Checkout Flow (Onsite Data Scientist)

Context: You will run an online experiment of a new checkout flow. The baseline conversion rate is 5%, and the product team expects a 3% relative lift (to 5.15%). You must select metrics, size the test, design randomization to avoid contamination, pre-register the analysis, and plan segmented readouts.

(a) Metrics

  • Choose one primary metric and at least two guardrail metrics; justify your choices.

(b) Sample Size

  • Compute the required sample size per group for 90% power at α = 0.05 (two-sided), given:
    • 10% of traffic will be excluded as bot traffic post-exposure.
    • A 5% day-of-week (DOW) effect (seasonality) must be accounted for.
  • State all assumptions used.

(c) Randomization & Bucketing

  • Propose a scheme that prevents contamination across sessions and devices.

(d) Pre-registration

  • Define a pre-registration plan, including whether you use a fixed-horizon or sequential design, and how peeking will be handled (e.g., group sequential or alpha spending).

(e) Segmentation & Multiplicity

  • Describe how you will segment results (e.g., new vs returning) while controlling for multiplicity.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Other•More Data Scientist•Other Data Scientist•Other Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.