How should Uber evaluate lower ETA?
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Uber wants to estimate the business value of reducing pickup ETA, where ETA refers to the rider-facing estimated time for a driver to arrive at the pickup location. Assume the change may affect both rider behavior and marketplace dynamics, so interference between riders and drivers is possible.
Answer the following:
1. What metrics would you track to evaluate the impact of lowering ETA?
- Include primary success metrics, downstream business metrics, guardrail metrics, and diagnostic metrics.
- Consider rider outcomes, driver outcomes, marketplace health, and financial impact.
2. From a business perspective, why is lowering ETA important?
- Discuss how ETA may affect conversion, cancellations, retention, supply-demand balance, and user trust.
- Also discuss possible tradeoffs or unintended consequences.
3. Suppose that in observational data you find a positive correlation between ETA and session conversion rate, where session conversion rate is defined as:
- `successful trip requests / app-open sessions`
How would you interpret this result?
- Why does correlation not necessarily imply a causal effect?
- What confounding variables, selection effects, or aggregation effects could produce this pattern?
4. How would you design a causal evaluation of lowering ETA?
- Compare when you would use a user-level A/B test, a geo-time switchback experiment, or a synthetic control approach.
- Explain the experimental unit, randomization strategy, analysis plan, and how you would handle marketplace spillovers.
5. After the experiment, suppose the 95% confidence interval for the treatment effect on session conversion rate is `[-5%, +1%]`.
- How would you interpret this result statistically and from a business decision perspective?
- What does it say about uncertainty, power, and practical significance?
- How would you improve precision in a future experiment?
6. Now assume the experiment was run as a user-level A/B test. During readout, the PM asks for a deep dive on users who took at least 5 trips during the experiment.
- How would you respond?
- What is problematic about conditioning on this subgroup after randomization?
- How would you communicate the risk of post-treatment bias or selection bias, and what alternative analyses would you recommend instead?
Quick Answer: This question evaluates competency in metrics design, causal inference, and experiment strategy for two-sided marketplaces, including detecting and addressing interference, interpreting observational correlations versus causal effects, and understanding statistical inference and subgroup selection bias.