Design and power an A/B test

Quick Overview

This question evaluates a data scientist's competency in experimentation design, including metrics selection and guardrails, statistical power and sample-size calculation, sequential monitoring and ramp strategies, operational checks for randomization and contamination, and choosing credible quasi-experimental alternatives.

Design and power an A/B test

Company: Stripe

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

You plan to launch a targeting model via email: treat = users above a score threshold receive an email; control = withheld. (1) Choose a primary success metric and two guardrails (e.g., unsubscribe rate, complaint rate) and justify them. (2) Given a baseline 7-day purchase rate of 5% and an expected relative lift of 8%, compute the minimum per-arm sample size for a two-sided test with α=0.05 and 80% power; show your formulas/assumptions (continuity-corrected normal approximation is fine). (3) Propose a ramp plan with sequential monitoring that controls type-I error (e.g., group-sequential or alpha-spending); specify interim looks and stopping rules. (4) Describe pre-experiment checks (randomization, covariate balance, holdout contamination) and how you'd handle interference and seasonality around weekends. (5) If legal or traffic constraints prevent pure A/B, propose a credible quasi-experimental design (e.g., regression discontinuity on the score threshold or staggered difference-in-differences), and list the assumptions you would test and the plots you'd include in your slides.

Quick Answer: This question evaluates a data scientist's competency in experimentation design, including metrics selection and guardrails, statistical power and sample-size calculation, sequential monitoring and ramp strategies, operational checks for randomization and contamination, and choosing credible quasi-experimental alternatives.

|Home/Analytics & Experimentation/Stripe
Stripe logo
Stripe
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
2
0

Email Targeting Model Experiment Design

You plan to launch a targeting model via email where:

  • Treatment: users above a score threshold receive an email
  • Control: eligible users are withheld

Answer the following:

  1. Metrics
    • Choose one primary success metric and two guardrails (e.g., unsubscribe rate, complaint rate). Justify each.
  2. Sample size
    • Baseline 7-day purchase rate: 5%
    • Expected relative lift: 8%
    • Compute the minimum per-arm sample size for a two-sided test with α = 0.05 and 80% power. Show formulas and assumptions. A continuity-corrected normal approximation is acceptable.
  3. Ramp and sequential monitoring
    • Propose a traffic ramp plan with sequential monitoring that controls type-I error (e.g., group-sequential or alpha-spending). Specify interim looks and stopping rules.
  4. Pre-experiment checks and operational risks
    • Describe checks for randomization, covariate balance, and holdout contamination. Explain how you would handle interference and seasonality around weekends.
  5. If pure A/B is infeasible
    • Propose a credible quasi-experimental design (e.g., regression discontinuity on the score threshold or staggered difference-in-differences). List key assumptions you would test and the plots you would include in slides.
Loading comments...