How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Stripe.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Stripe during technical interviews.

Design and power an A/B test | Stripe Interview Question

Quick Overview

This question evaluates a data scientist's competency in experimentation design, including metrics selection and guardrails, statistical power and sample-size calculation, sequential monitoring and ramp strategies, operational checks for randomization and contamination, and choosing credible quasi-experimental alternatives.

Email Targeting Model Experiment Design

You plan to launch a targeting model via email where:

Treatment: users above a score threshold receive an email
Control: eligible users are withheld

Answer the following:

Metrics
- Choose one primary success metric and two guardrails (e.g., unsubscribe rate, complaint rate). Justify each.
Sample size
- Baseline 7-day purchase rate: 5%
- Expected relative lift: 8%
- Compute the minimum per-arm sample size for a two-sided test with α = 0.05 and 80% power. Show formulas and assumptions. A continuity-corrected normal approximation is acceptable.
Ramp and sequential monitoring
- Propose a traffic ramp plan with sequential monitoring that controls type-I error (e.g., group-sequential or alpha-spending). Specify interim looks and stopping rules.
Pre-experiment checks and operational risks
- Describe checks for randomization, covariate balance, and holdout contamination. Explain how you would handle interference and seasonality around weekends.
If pure A/B is infeasible
- Propose a credible quasi-experimental design (e.g., regression discontinuity on the score threshold or staggered difference-in-differences). List key assumptions you would test and the plots you would include in slides.

Quick Overview

Email Targeting Model Experiment Design

You plan to launch a targeting model via email where:

Treatment: users above a score threshold receive an email

Control: eligible users are withheld

Answer the following:

Metrics

Choose one primary success metric and two guardrails (e.g., unsubscribe rate, complaint rate). Justify each.

Sample size

Baseline 7-day purchase rate: 5%
Expected relative lift: 8%
Compute the minimum per-arm sample size for a two-sided test with α = 0.05 and 80% power. Show formulas and assumptions. A continuity-corrected normal approximation is acceptable.

Ramp and sequential monitoring

Propose a traffic ramp plan with sequential monitoring that controls type-I error (e.g., group-sequential or alpha-spending). Specify interim looks and stopping rules.

Pre-experiment checks and operational risks

Describe checks for randomization, covariate balance, and holdout contamination. Explain how you would handle interference and seasonality around weekends.

If pure A/B is infeasible

Propose a credible quasi-experimental design (e.g., regression discontinuity on the score threshold or staggered difference-in-differences). List key assumptions you would test and the plots you would include in slides.

Design and power an A/B test

Quick Overview

Email Targeting Model Experiment Design

Solution

Submit Your Answer to Earn 20XP

Design and power an A/B test

Quick Overview

Email Targeting Model Experiment Design

Solution

Submit Your Answer to Earn 20XP