Evaluate New Model's Impact on Rider and Driver Experience
Overview
This question evaluates experimental design, metric selection, behavioral analytics, and causal inference skills for measuring a new ETA model’s impact on rider and driver experience within the Analytics & Experimentation domain.
Uber
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Airport Pickups ETA Model: Evaluation and Experiment Design
Context
A new model predicts rider walk-time from the order location (e.g., terminal/door) to the pickup zone. The current baseline ETA shown to riders reflects only the driver’s arrival. The product will display a combined pickup ETA (driver ETA ± walk time), which can change rider behavior: riders may speed up or slow down based on the displayed time.
Assumptions added for clarity:
- You can log rider device location pings to infer arrival at pickup zones and approximate walking start/arrival times.
- You can log driver arrival/departure times at pickup zones.
- Both baseline and new model predictions can be logged, regardless of which is shown to the rider.
Tasks
- Metrics: What would you use to evaluate the new airport ETA model’s impact on rider and driver experience?
- Experiment: Design an experiment to test the new model against the baseline.
- Feedback loop: Riders may adjust their pace based on displayed ETA. How would you quantify this behavioral response and separate it from pure model accuracy?
Solution
Login required
Show