Evaluate Promotion Campaign Effectiveness with A/B Testing
Overview
This question evaluates a data scientist's skills in experimental design, A/B testing methodology, KPI and guardrail selection, power and sample-size calculations, MDE interpretation, and statistical communication.
Uber
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
55
0
A/B Testing a San Francisco–Only January Promotion
Background
A consumer marketplace app plans to run a January 2024 promotion limited to San Francisco users. The team will evaluate effectiveness via an A/B experiment.
Tasks
-
Business Rationale
- Why might the company want to launch this promotion in January and limit it to San Francisco?
-
Success Metrics
- Define the primary KPI(s) and key secondary/guardrail metrics you would monitor to judge success.
-
Sample Size and Power
- Describe how you would determine the required sample size for the A/B test.
- Which inputs and factors influence sample size and test duration?
-
Minimum Detectable Effect (MDE)
- Revenue is the primary metric, and the product manager sets the MDE to 0.5% (relative). Interpret this value and explain how it affects sample-size calculations.
-
Result Interpretation
- Suppose the observed revenue lift is 0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager?
Notes
Focus on goal alignment, KPI selection, power analysis, variance, effect size, significance level, and clear communication of non-significant but directional results.
Solution
Login required
Show