Interpret a Smart Wait Launch with Conflicting Metrics
Company: Waymo
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Take-home Project
A product changes its wait-time display to show more conservative estimates. After launch, conversion falls by 5% while actual time to pickup falls by 25%. A stakeholder concludes that fleet efficiency improved. Do you agree? Give alternative explanations for the pickup-time decrease and explain how the conversion change affects your interpretation.
### Constraints & Assumptions
- The figures are observed before-after changes, not necessarily estimates from a randomized experiment.
- Conversion and time to pickup may be measured on different denominators.
- The display change can affect which users choose to proceed, even if fleet operations do not change.
### Clarifying Questions to Ask
- How are conversion and time to pickup defined, and what are their denominators?
- Is time to pickup measured only for completed or accepted rides?
- Was the launch randomized or simultaneous with operational, geographic, or demand changes?
- Did rider mix, trip mix, supply, pricing, cancellations, or logging change?
- Is the 5% conversion drop relative or absolute, and what uncertainty surrounds both estimates?
### What a Strong Answer Covers
- Rejection of a causal fleet-efficiency claim from the two aggregate movements alone.
- Selection or composition effects caused by discouraging users with longer expected waits.
- Other hypotheses such as supply-demand shifts, routing changes, geographic mix, seasonality, or measurement changes.
- A metric framework that considers conversion, pickup time, completed trips, cancellations, utilization, reliability, and user experience together.
- A credible randomized or phased test with stable metric definitions, guardrails, segmentation, and uncertainty.
- A decision recommendation that distinguishes display accuracy from actual operational improvement.
### Follow-up Questions
1. What would happen to observed pickup time if mostly long-wait users stopped converting?
2. Which metric would more directly measure estimate accuracy?
3. How would you design the randomization unit?
4. What segment analysis would you pre-specify?
Quick Answer: Interpret a product launch where conservative wait estimates reduce conversion while observed pickup time also falls. Diagnose selection effects and alternative causes, define customer and fleet metrics with consistent denominators, and propose an experiment that can separate display accuracy from operational improvement.