Measuring a New Uber Campaign's Incremental Impact with a Controlled Experiment
Company: Uber
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
Uber is getting ready to launch a new campaign and wants evidence that it actually works before rolling it out widely. How would you design an A/B test to evaluate it?
```hint Look beyond the treated user
Riders and drivers in a city draw on the same pool of trips. Ask yourself whether giving the campaign to some users could change what control users in the same city experience, and what that means for how you split traffic.
```
### Clarifying Questions
- What kind of campaign is this: a rider promotion such as ride credits, a driver incentive, or a message like a push notification or email, and is it for Rides or Eats?
- What is it meant to achieve: acquire new riders, win back lapsed ones, or increase trip frequency among active riders?
- Is there one version of the campaign, or several variants (offer sizes, message copy) to compare?
- Is there a fixed budget or launch date that limits how long the test can run?
### What a Strong Answer Covers
- A testable hypothesis and a primary metric tied to the campaign's goal, with guardrails for cost and marketplace health
- A unit of randomization chosen with spillover between riders and drivers in mind, and the trade-off that choice carries
- Sample size, power and duration, including weekly cycles, novelty effects and trips pulled forward from later weeks
- An intent-to-treat analysis plan, handling of multiple variants, and a launch decision rule agreed in advance
- Validity checks and failure modes such as sample ratio mismatch, contamination and overlapping campaigns
### Follow-up Questions
- The lift is strong in the first week and fades over the following weeks. How do you interpret that, and what do you recommend?
- Trips rose in the treatment group, but control riders in the same cities saw longer pickup times. What does that say about your estimate, and how would you redesign the test?
- Finance asks whether the campaign is worth what it costs. How do you turn the test result into a cost per incremental trip?
Overview: A product manager experiment-design question about evaluating a new Uber campaign with a controlled A/B test. It tests choosing a primary metric and guardrails, picking a randomization unit for a two-sided marketplace, sizing and timing the test, and setting a clear launch decision rule.