Metric Diagnosis Case: Investigating a 15 Percent Jump in Cancelled Orders

Order cancellations on your platform have gone up by 15 percent. As the product manager, explain how you would find the root cause: what you would check first, how you would break the problem down, and how you would decide which explanation is actually responsible before recommending a fix. ```hint Narrow before you brainstorm An order can be cancelled by different parties and at different points in its life. Ask which of those cuts would shrink the search fastest, and whether the number itself can be trusted. ``` ### Clarifying Questions - Is the 15 percent a rise in the number of cancelled orders or in the cancellation rate, and is it relative (for example 10 percent of orders becoming 11.5 percent) or in percentage points? - What kind of orders are these, such as restaurant delivery, retail e-commerce or online checkout for merchants, and who is able to cancel one: the customer, the seller, a courier, or the system itself? - Over what period did the increase happen, and was it a step change on a particular day or a gradual climb? - Did related numbers move at the same time, such as total orders, checkout conversion, delivery times or support contacts? ### What a Strong Answer Covers - Checking the metric before the product: its definition, recent logging or pipeline changes, and whether a count or a rate moved. - A structured decomposition of cancellations by initiator, order stage and time to cancel, and by segments such as region, platform, app version, seller and payment method. - Internal causes (releases, fee or policy changes, operational capacity) separated from external ones (seasonality, competitors, partner outages), with hypotheses ranked by how well they fit the data. - Adjacent metrics used to tell a harmful rise from a benign one, and a plan to confirm the leading hypothesis. - A sized impact and a concrete next step or mitigation once the cause is found. ### Follow-up Questions - The whole increase comes from one app version on one platform. What do you do in the next day, and what afterwards? - Cancellations went up but completed orders per day did not change. Is this a problem, and how would you decide? - How would you set up monitoring so a rise like this is caught within days instead of months?