Evaluate Smart Wait Launch Impact
Company: Waymo
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Online Assessment
A ride-hailing or autonomous-vehicle product launches a feature called **Smart Wait**. Before a user confirms a ride, Smart Wait shows a more conservative estimated wait time, meaning the displayed estimate is usually longer than before. The goal is to improve ETA accuracy and reduce user disappointment from overly optimistic wait estimates.
After launch, the team observes:
- Quote-to-ride conversion rate decreased by 5% relative.
- Actual time to pickup, abbreviated TTP, decreased by 25% among completed rides.
A product manager claims: "Fleet efficiency improved because actual TTP decreased."
Answer the following:
1. Do you agree with the PM's conclusion? Why or why not?
2. What alternative hypotheses could explain the TTP decrease, especially in relation to the conversion drop?
3. What metrics and analyses would you use to decide whether Smart Wait was beneficial overall?
4. If you were designing an experiment for this feature, what would you randomize, what would be the primary metric, and what guardrails would you monitor?
Overview: This question evaluates a data scientist's competencies in causal inference and experiment analysis, including interpreting A/B test outcomes, trade-offs between conversion and operational metrics, selection of primary and guardrail metrics, and generation of alternative hypotheses.
Community answers
Answer by pai.b.pradeep
Short answer:
No, I wouldn't accept the PM's conclusion yet. A lower time-to-pickup (TTP) alongside a conversion drop is a classic
selection effect
. Smart Wait may have filtered out the riders who would have had long waits, rather than making any individual wait shorter.
> Assumption: the setup is that Smart Wait launched, average TTP fell, ride-request conversion fell, and the PM is calling it a win. Adjust if your prompt differs.
Do I agree?
Not without more evidence. TTP is measured only on
completed
rides, so its denominator changed. If the users who dropped off were the ones facing long waits, average TTP falls mechanically even if nobody's experience improved. Also check whether this was a before/after comparison, which would leave it exposed to seasonality, fleet-size changes, and other launches.
> 2. Alternative hypotheses
> Survivorship/mix shift:
long-wait requests (far pickups, peak hours, edge zones) now abandon, which lowers the average.
> Display effect:
Smart Wait shows longer or more honest ETAs, so users bail before requesting.
> Supply side:
fewer requests free up vehicles, which genuinely shortens waits for the riders who remain. This is a real benefit, but it comes at the cost of lost trips.
> External factors:
fleet changes, demand seasonality, or geography mix.
> 3. How to judge overall benefit
Measure TTP on an
> intent-to-treat
basis: compare all sessions, not just completed trips.
> Decompose
the TTP change into a within-segment effect and a mix effe