Design Pricing Model Experiment
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You work as a data scientist for a ride-hailing marketplace. The company wants to launch a new pricing model that may change the price shown to riders and the earnings or trip value shown to drivers in the request prompt.
Design an experiment to evaluate whether the new pricing model should be launched.
Address the following questions:
1. Before designing the experiment, how would you reason through the driver's mental state when receiving a trip request prompt?
2. What primary metrics and guardrail metrics would you use for riders, drivers, and the marketplace?
3. If the experiment is run in only one city using ordinary treatment and control groups, what problems can arise from marketplace network effects or interference?
4. How would you design a switchback experiment for one city?
5. How would you handle limited sample size in a switchback test?
6. What are the tradeoffs of using full-day switchback windows in one city?
7. What are the tradeoffs between larger and smaller treatment groups?
8. What are the tradeoffs between longer and shorter treatment windows?
9. If the company later rolls the test out to 12 cities, how would you increase sample size and improve the experimental design?
Quick Answer: This question evaluates experimental design and causal inference competencies for marketplace pricing, covering metric selection, guardrail definition, interference and switchback designs, and sample-size and treatment-window tradeoffs.