Evaluate a cold-start rating launch
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Uber Eats is considering showing an initial rating for newly onboarded restaurants that have little or no historical review data. This is a two-sided marketplace: diners choose restaurants, couriers choose deliveries, and both demand and supply can shift when visibility changes.
Answer the following:
1. What business benefits could this feature create for the platform, diners, couriers, and merchants?
2. What costs or risks could it introduce? Consider marketplace liquidity, courier pickup times, incentive spend, misranking, and merchant fairness.
3. Suppose someone proposes a standard user-level A/B test where 5 percent of users do not see the new restaurant while 95 percent do. Is this a valid design? Why or why not?
4. If not, propose a better experiment design for a two-sided marketplace with interference.
5. After launching the feature, how would you estimate the true incremental business impact, especially if you cannot randomize forever? Discuss a holdout or geo-based counterfactual approach and the assumptions required.
Quick Answer: This question evaluates a data scientist's competency in marketplace analytics, causal inference, experimentation design and measurement, specifically around handling interference, assessing incremental impact, and weighing trade-offs between visibility, liquidity, and fairness.