Lyft Metric Investigation: Root-Causing a Sudden Jump in Cancelled Trips

Quick Overview

A metric diagnosis case for a Lyft product manager: ride cancellations jumped this week, and the candidate must explain what could cause it and how to confirm the real cause. It tests metric decomposition, hypothesis generation across internal and external factors, and putting data checks in a sensible order.

Lyft Metric Investigation: Root-Causing a Sudden Jump in Cancelled Trips

Company: Lyft

Role: Product Manager

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Other

You are a product manager at Lyft. Ride cancellations rose sharply this week compared with a normal week. What could explain the jump, and how would you work out which explanation is the real one? ```hint Who cancelled, and at what point A ride can be cancelled by the rider or the driver, before a driver is matched, while the driver is on the way, or at pickup. Break the metric down along these lines before you guess at causes. ``` ### Clarifying Questions - Is the increase in the number of cancellations or in the cancellation rate per ride request, and did requests themselves change this week? - Does the metric count rider cancellations, driver cancellations and requests that ended with no driver found together, or separately? - Is the rise concentrated in certain cities, ride types, platforms or app versions, and did it start on one day or build through the week? ### What a Strong Answer Covers - Confirming the metric's definition and ruling out a logging or data-pipeline cause before anything else - A decomposition by who cancelled, the stage of the trip, and segments such as market, ride type, platform, app version and rider or driver tenure - Hypotheses spanning internal changes (releases, matching, pricing, policies, driver pay) and external ones (weather, events, competitors, driver supply, outages), each with the data that would confirm or reject it - A sensible order for the checks, and how the impact on completed rides is sized - Next steps once the cause is found, with guardrail metrics and monitoring ### Follow-up Questions - The breakdown shows the rise comes from drivers cancelling after they accept, mostly on long pickups in two cities. What do you do next? - Cancellations rose but completed rides did not fall. Is this still a problem? - How would you set up monitoring so the next spike is detected and attributed within a day?

Overview: A metric diagnosis case for a Lyft product manager: ride cancellations jumped this week, and the candidate must explain what could cause it and how to confirm the real cause. It tests metric decomposition, hypothesis generation across internal and external factors, and putting data checks in a sensible order.

Lyft logo
Lyft
Aug 4, 2024
mediumProduct ManagerOtherAnalytics & Experimentation
0
0

You are a product manager at Lyft. Ride cancellations rose sharply this week compared with a normal week. What could explain the jump, and how would you work out which explanation is the real one?

Clarifying Questions Guidance

  • Is the increase in the number of cancellations or in the cancellation rate per ride request, and did requests themselves change this week?
  • Does the metric count rider cancellations, driver cancellations and requests that ended with no driver found together, or separately?
  • Is the rise concentrated in certain cities, ride types, platforms or app versions, and did it start on one day or build through the week?

What a Strong Answer Covers Guidance

  • Confirming the metric's definition and ruling out a logging or data-pipeline cause before anything else
  • A decomposition by who cancelled, the stage of the trip, and segments such as market, ride type, platform, app version and rider or driver tenure
  • Hypotheses spanning internal changes (releases, matching, pricing, policies, driver pay) and external ones (weather, events, competitors, driver supply, outages), each with the data that would confirm or reject it
  • A sensible order for the checks, and how the impact on completed rides is sized
  • Next steps once the cause is found, with guardrail metrics and monitoring

Follow-up Questions Guidance

  • The breakdown shows the rise comes from drivers cancelling after they accept, mostly on long pickups in two cities. What do you do next?
  • Cancellations rose but completed rides did not fall. Is this still a problem?
  • How would you set up monitoring so the next spike is detected and attributed within a day?