Metrics and diagnosing a drop

Metrics questions in a PM interview ask how you'd know a product works, and metric drop questions ask why a number fell and what you'd do about it. I'll use one generic product for both: a dog-walking app where owners book local walkers. The second half answers this prompt:

"Weekly bookings on a dog-walking app are down 12% over six weeks. What happened?"

How to choose metrics for a product

Before naming any metric, write what a person would do differently if the product worked:

"An owner who used to scramble for a neighbor at lunchtime now books a walker without thinking about it, and books again next week. A walker fills their week close to home."

Then build outward from that sentence.

One main number. Pick the count closest to that behavior: completed walks per week. I prefer it to bookings, since a cancelled booking gave nobody anything, and to downloads, since a download walks no dogs. Choosing one forces a priority when two good numbers disagree.

The inputs that move it. How many owners are active, how often each asks for a walk, how often a request finds a walker. A team can work on these directly within a quarter, and the main number moves only through them.

The guardrail. A guardrail is a number you promise not to damage while chasing the main one. Completed walks can rise because you cut prices and paid for it out of walker pay, until walkers leave. So I'd watch walker earnings per hour, and lost-dog or injury reports per 1,000 walks.

For the case below, write the behavior sentence and one main number, then stop.

Why a change over time beats an all-time total

A total since launch can only go up. Assume the app has handled 2,000,000 bookings since launch, and weekly bookings slid from 10,000 to 8,800 over six weeks. At an average of about 9,400 a week, 9,400 × 6 is about 56,000 more, so the total reads about 2,056,000. That chart still climbs while the business shrinks. Cracking the PM Interview also tells you to watch the change over the total.

I'd add two habits. Use rates where you can, such as bookings per active owner, so growth from new people can't hide a decline. And group users by when they joined, which analysts call a cohort. If owners who joined in April keep booking less at week eight than owners who joined in March did, trouble is coming even while totals rise.

Which data source answers which question

That last limit bites on a marketplace, because test and control owners share the same dog walkers. Assume a city's walkers have time for 1,000 walks a week, split evenly at 500 per group. A new feature lets test owners book faster, and they take slots first: test goes to 560, control falls to 440. The test looks 27% better (560 ÷ 440 ≈ 1.27), yet the total is still 1,000. The feature moved walks between groups and would add nothing at full rollout.

The usual fix is to split by city or neighborhood so the groups don't share walkers, or to switch the feature on and off by week. Either way you have fewer independent groups to compare, so the test takes longer.

For the case below, write only what you'd randomize and why, then stop.

Diagnosing a metric drop: is the number real?

Now the prompt. Before inventing causes, spend a minute on the number itself. First ask whether "bookings" means made or completed, and offer a default. Then check two things.

Did the measurement change? A new app version that logs bookings differently can shrink a number when nothing happened. So I'd compare with a source that doesn't depend on app tracking, such as walker payouts. If payouts fell by about 12% too, the drop is real.

Is the comparison fair? Pet care has busy seasons, with the holidays among the busiest, so look at the same six weeks last year. If last year's same weeks fell 10%, only about 2 points of this year's 12 are new.

For this example, suppose payouts fell 12% and last year was flat. It's real.

For the case below, write only your checks on whether the drop is real, then stop.

Was the drop sudden or gradual?

Ask for the weekly chart. A cliff in one week points to something with a date on it: a release, a price change, an outage. A slope points to something building up, such as walkers drifting away, a rival growing, or newer owners not sticking.

Two traps sit here. A release that reached users over several weeks draws a slope, so a slope doesn't clear your own team. And a gentle slope in the total can be a steep one in a single city, averaged with flat lines everywhere else.

Suppose this chart slopes steadily. I'd lean toward causes that build over time, and keep that second trap in mind.

Write bookings as a formula

A total can only fall if one of its parts fell, so write the parts down. For this app:

Weekly bookings = active owners × requests per owner × share of requests that find a walker

Read the units across. Owners times requests per owner gives requests, and requests times the share filled gives bookings. The last term matters here because it brings walkers in, and a marketplace can break on either side.

These numbers are my assumptions. Six weeks ago: 5,000 owners × 2.5 requests = 12,500 requests, and 12,500 × 80% = 10,000 bookings. Today there are 8,800. Suppose the logs show owners still at 5,000 and requests still at 2.5 each. Then the share filled must be 8,800 ÷ 12,500 = 70.4%.

Owners are still asking, and fewer requests find a walker. Had owners been the term that moved, I'd split them next into new and returning.

For the case below, write the formula, check its units and stop.

Slice the metric to find where it lives

Cut the term that moved along one dimension at a time: city, weekday or weekend, time of day, new or long-time walkers, app version. For each cut, compare a group's share of the drop with its share of the volume.

Suppose one metro has 3,750 of the 12,500 requests, which is 30%. Its filled requests went from 3,750 × 80% = 3,000 down to 1,800, so 1,800 ÷ 3,750 = 48%. Everywhere else, 8,750 × 80% = 7,000, unchanged. And 1,800 + 7,000 = 8,800, so that's the whole drop: 30% of the requests and all 1,200 lost bookings.

Slice again inside the metro. Suppose active walkers there fell from 400 to 240, which is 40%, the same as the fall from 3,000 to 1,800. Stop cutting once a new cut stops narrowing the cause.

For the case below, name the slices you'd try first and stop before any causes.

Rank causes before checking them

Why did walkers leave that metro? List what could explain it, then order the list by how well each fits the evidence and how cheaply you can check it.

Ranking first keeps you from checking whatever came to mind first, and it gives the interviewer something to react to. If they say "no rival has entered," cross it off out loud and go to the next row.

For the case below, list your causes and rank them, then stop before checking any.

What to say at the end

Finish with where the drop lives, what you think caused it, how sure you are, and what happens next:

"The 12% looks real: walker payouts fell by the same amount, and the same weeks last year were flat. All of it sits in one metro, where the share of requests that find a walker fell from 80% to 48%. Owners are still asking, so this is a walker problem. My leading guess is a rival paying more, which I haven't confirmed. On Monday I'd call ten walkers who stopped, and I'd offer a time-limited bonus for unfilled slots in that metro, watching cost per extra booking so we don't buy walks at a loss. If the calls point elsewhere, the bonus stops and I go to the next cause on my list."

Saying "I haven't confirmed" rarely costs you anything here. The interviewer can see you've had no way to check, and naming a cause with full confidence on thin data is the bigger risk. The bonus gets an end date for the same reason a launch gets a result that would make you stop: the limit is easier to set before anyone is attached to the spending.

For the case below, work out your own formula and causes from scratch rather than reusing the walker story, then give your closing out loud, Monday included, in under a minute.