Design Pricing Model Experiment
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You work as a data scientist for a ride-hailing marketplace. The company wants to launch a new pricing model that may change the price shown to riders and the earnings or trip value shown to drivers in the request prompt.
Design an experiment to evaluate whether the new pricing model should be launched.
Address the following questions:
1. Before designing the experiment, how would you reason through the driver's mental state when receiving a trip request prompt?
2. What primary metrics and guardrail metrics would you use for riders, drivers, and the marketplace?
3. If the experiment is run in only one city using ordinary treatment and control groups, what problems can arise from marketplace network effects or interference?
4. How would you design a switchback experiment for one city?
5. How would you handle limited sample size in a switchback test?
6. What are the tradeoffs of using full-day switchback windows in one city?
7. What are the tradeoffs between larger and smaller treatment groups?
8. What are the tradeoffs between longer and shorter treatment windows?
9. If the company later rolls the test out to 12 cities, how would you increase sample size and improve the experimental design?
Overview: This question evaluates experimental design and causal inference competencies for marketplace pricing, covering metric selection, guardrail definition, interference and switchback designs, and sample-size and treatment-window tradeoffs.
Community answers
Answer by dyeshu79
To reason through the driver's mental model, they evaluate the trade-off between time, distance, and earnings. Drivers are much more likely to accept simpler, shorter trips with fair pay. Conversely, for difficult trips involving dense traffic or longer routes, securing an adequate fare is harder, making drivers hesitant to take the risk.
North Star Metric: Number of completed rides. This helps track whether drivers continue using the platform when the pricing model changes.
Guardrail Metrics: Driver churn remains flat, the active driver pool remains stable, and market share is protected against cross-platform competitors.
Interference & Marketplace Challenges:
a. Cannibalization: This is a major issue. For example, if drivers in the treatment group complete fewer rides, it directly benefits the control group by opening up excess demand.
b. Scalability: An experiment might reach statistical significance locally, but the pricing model may only work in specific regions, creating a scalability barrier.
c. Violation of Independence: Standard A/B testing assumes independent units (SUTVA). In a two-sided marketplace, riders and drivers interact in a shared resource pool, making cross-group cannibalization a significant risk.
We can use a switchback testing system, where the entire city alternates between the treatment and control environments over set time blocks.
We can apply techniques like CUPED to reduce pre-experiment variance using historical user-level data. Additionally, we