Determine Sample Size for Promotion Campaign A/B Test

Q: Determine Sample Size for Promotion Campaign A/B Test

This question evaluates competency in experimental design, statistical power and sample size estimation, metric selection and guardrail definition, and interpretation of A/B test outcomes for revenue-focused promotions.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Design for a Threshold Discount Promotion

Scenario

A company plans to launch a threshold-based promotion (e.g., "20% off when spending $40"). The team wants to evaluate the impact with an A/B test. Revenue will be the primary success metric.

Assumptions (to make the task self-contained):

Randomization is at the user level.
The metric will be measured over a fixed analysis window (e.g., 14 days per user) to dampen day-to-day volatility.
Standard significance level (alpha) = 0.05 (two-sided) and power = 0.80 unless otherwise stated.

Tasks

Why might the business introduce this promotion? State plausible objectives.
Define success metrics: primary, secondary, and guardrails.
How would you determine the required sample size? What inputs and assumptions are needed?
The PM sets the minimum detectable effect (MDE) to 0.5% on revenue. Interpret this value and explain how it influences sample size.
If the observed revenue lift is 0.3% and not statistically significant, how would you interpret and communicate this result to the PM?

Determine Sample Size for Promotion Campaign A/B Test

A/B Test Design for a Threshold Discount Promotion

Scenario

Tasks

Solution

Comments (0)

Determine Sample Size for Promotion Campaign A/B Test

Overview

A/B Test Design for a Threshold Discount Promotion

Scenario

Tasks

Solution

Comments (0)