##### Scenario Uber plans to launch a promotion campaign and wants to evaluate its effectiveness with an A/B experiment. Two concrete variants of the offer have come up in interviews: - A spend-threshold discount (e.g., **20% off when a user spends $40**), and - A geo-targeted promo limited to a single market (e.g., a **January-2024 campaign available only to San-Francisco users**). Treat the offer as a generic promotion; the reasoning below applies to either framing. ##### Question 1. **Business rationale.** Why might the company want to launch this promotion campaign? (If geo-limited, why restrict it to a single market like San Francisco?) 2. **Success metrics.** Which business metrics would you monitor to judge success? Distinguish primary, secondary/diagnostic, and guardrail metrics. 3. **Sample size.** How would you determine the required sample size for the study? What inputs, assumptions, and factors influence it? 4. **Interpreting the MDE.** Revenue is chosen as the primary metric and the product manager sets the minimum detectable effect (MDE) to **0.5%**. How do you interpret this value and how does it affect the sample-size calculation? 5. **Non-significant result.** Suppose the observed lift is **+0.3%** but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager? ##### Hints Think about goal alignment, primary/secondary/guardrail KPIs, baseline mean and variance, power and Type-I/Type-II errors, effect size, significance level, variance reduction (e.g., CUPED), and how to explain a small, directional-but-non-significant effect to a stakeholder.

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Sample Size for Promotion Campaign A/B Test states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Determine Sample Size for Promotion Campaign A/B Test

Scenario

Uber plans to launch a promotion campaign and wants to evaluate its effectiveness with an A/B experiment. Two concrete variants of the offer have come up in interviews:

A spend-threshold discount (e.g., 20% off when a user spends $40 ), and
A geo-targeted promo limited to a single market (e.g., a January-2024 campaign available only to San-Francisco users ).

Treat the offer as a generic promotion; the reasoning below applies to either framing.

Question

Business rationale. Why might the company want to launch this promotion campaign? (If geo-limited, why restrict it to a single market like San Francisco?)
Success metrics. Which business metrics would you monitor to judge success? Distinguish primary, secondary/diagnostic, and guardrail metrics.
Sample size. How would you determine the required sample size for the study? What inputs, assumptions, and factors influence it?
Interpreting the MDE. Revenue is chosen as the primary metric and the product manager sets the minimum detectable effect (MDE) to 0.5% . How do you interpret this value and how does it affect the sample-size calculation?
Non-significant result. Suppose the observed lift is +0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager?

Hints

Think about goal alignment, primary/secondary/guardrail KPIs, baseline mean and variance, power and Type-I/Type-II errors, effect size, significance level, variance reduction (e.g., CUPED), and how to explain a small, directional-but-non-significant effect to a stakeholder.

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?