##### Scenario The business is launching a January-2024 promotion campaign available only to San-Francisco users and wants to evaluate its effectiveness through an A/B experiment. ##### Question Why do you think the company wants to launch this promotion campaign? What business metrics would you monitor to judge success? How would you determine the required sample size for the study? Which factors influence it? Revenue is chosen as the primary metric for the A/B test and the product manager sets the minimum detectable effect (MDE) to 0.5%. How do you interpret this value and how does it affect sample-size calculations? Suppose the observed lift is 0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager? ##### Hints Discuss goal alignment, primary/secondary KPIs, power analysis, variance, effect size, significance level, and ways to explain non-significant but directional results.

This question evaluates a data scientist's skills in experimental design, A/B testing methodology, KPI and guardrail selection, power and sample-size calculations, MDE interpretation, and statistical communication.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Evaluate Promotion Campaign Effectiveness with A/B Testing

A/B Testing a San Francisco–Only January Promotion

Background

A consumer marketplace app plans to run a January 2024 promotion limited to San Francisco users. The team will evaluate effectiveness via an A/B experiment.

Tasks

Business Rationale
- Why might the company want to launch this promotion in January and limit it to San Francisco?
Success Metrics
- Define the primary KPI(s) and key secondary/guardrail metrics you would monitor to judge success.
Sample Size and Power
- Describe how you would determine the required sample size for the A/B test.
- Which inputs and factors influence sample size and test duration?
Minimum Detectable Effect (MDE)
- Revenue is the primary metric, and the product manager sets the MDE to 0.5% (relative). Interpret this value and explain how it affects sample-size calculations.
Result Interpretation
- Suppose the observed revenue lift is 0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager?

Notes

Focus on goal alignment, KPI selection, power analysis, variance, effect size, significance level, and clear communication of non-significant but directional results.

Tasks

Business Rationale

Why might the company want to launch this promotion in January and limit it to San Francisco?

Success Metrics

Define the primary KPI(s) and key secondary/guardrail metrics you would monitor to judge success.

Sample Size and Power

Describe how you would determine the required sample size for the A/B test.
Which inputs and factors influence sample size and test duration?

Minimum Detectable Effect (MDE)

Revenue is the primary metric, and the product manager sets the MDE to 0.5% (relative). Interpret this value and explain how it affects sample-size calculations.

Result Interpretation

Suppose the observed revenue lift is 0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager?

Evaluate Promotion Campaign Effectiveness with A/B Testing

Quick Overview

A/B Testing a San Francisco–Only January Promotion

Background

Tasks

Notes

Solution

Comments (0)

Evaluate Promotion Campaign Effectiveness with A/B Testing

Quick Overview

A/B Testing a San Francisco–Only January Promotion

Background

Tasks

Notes

Solution

Comments (0)