1.2 Designing Metrics That Survive Scrutiny

Choosing a product metric means deciding which outcome a team should optimize. Several candidates may be reasonable but encourage different behavior. Imagine choosing a metric for the next four quarters: you need to explain what it rewards, who it counts, and what a favorable movement would mean. This lesson develops five tests for that choice, compares averages with threshold measures, and examines when sensitivity to outliers is useful. The previous lesson improved a given metric; here you decide whether that metric describes the outcome you actually want.

Why this matters in interviews

The question shows up in about five disguises, and they are all the same question.

• "You are launching a new product. Pick two metrics to watch for the first six months."

• "Between metric A and metric B, which would you use and why."

• "What is the north star for this product, and what would you watch alongside it."

• "How would you measure the performance of the support organization."

• "For a free-to-play game, do you expect mean revenue per player to sit above or below the median."

A definition tells the reader how a metric is calculated. A defensible choice also explains the behavior that optimizing it would reward. For each candidate metric below, ask how a team could improve the number while leaving users no better off. That question connects the arithmetic to the incentives created by the metric.

The moment a number goes on a dashboard with a target next to it, several hundred people begin steering toward it. Every gap between what the number measures and what the company wants gets found and exploited. Almost nobody involved intends to cheat. They are just doing the cheapest thing that moves the number they are judged on. So a senior candidate picks metrics the way you would write a contract with a counterparty who is smarter than you and not on your side.

The weak answer to the launch question, which I hear constantly, is "I would track daily active users and retention." That is not wrong so much as empty. There is no population, no window, no numerator, and no reason a team would change what it builds based on either number moving three points.

The stronger version sounds like this.

"I would track one acquisition metric and one engagement metric. I would define each so that a bot farm creating fake accounts makes it go down, not up. For acquisition, new accounts per day that complete the core action within 48 hours. For engagement, the share of 28-day-active accounts that complete the core action at least three times in a rolling week. The first is a volume number and the second is a rate, so together they catch both a growth stall and a quality collapse."

Interview tip: Name the adversary before you name the metric. Saying "here is how a team could hit this target without helping the product, and here is the change that closes it" is the single clearest senior signal in a metrics question.

---

The five tests a metric has to pass

Run every candidate through these five. In an interview, say them out loud as a checklist and then apply them to your own proposal. It converts a taste argument into an evaluation.

Tied to a decision

The test is a sentence, not a philosophy: name the decision. If weekly cook rate rises four points, does anyone staff differently, ship differently, or spend differently? If the honest answer is no, you have a report, not a metric.

This is where vanity metrics die. Cumulative registered accounts cannot go down, so its level is dominated by history and keeps climbing while the business stalls. Sign-ups can halve and the running total still goes up and to the right. All of the information lives in the increment, so report new accounts per period rather than the running sum. Total pageviews is worse, because it can be raised by making content harder to find.

Sensitive enough to see the effect you care about

A metric you cannot move in a readable amount of time is not a target, it is a mood. This is the test candidates skip, and it is the one with actual arithmetic behind it, so it is where you can separate yourself.

The arithmetic has a name: the minimum detectable effect, or MDE. It is the smallest true difference between two arms of an experiment that your test would actually catch, given the sample you can afford. "Catch" needs two numbers pinned down, and convention fixes both. Power of 80 percent means that if a true effect of exactly that size exists, you find it four times out of five. A 5 percent two-sided test means that when there is genuinely no effect at all, you wrongly declare one in 5 percent of experiments, counting mistakes in both directions. Quoted as a relative number, an MDE of 61 percent means that any real improvement smaller than 61 percent of the current level will usually come back reading flat.

Two things make the MDE what it is. First, it is a property of the metric's noise relative to its own level, not a property of your product. A quiet metric sitting at a healthy baseline has a small MDE. A wild one has a huge one, on the same users, in the same experiment. Second, it shrinks like one over the square root of the sample, so buying half the MDE costs four times the traffic. That combination is why the choice of metric, made once at the start, does more for your detection power than anything you can do afterwards.

We will price this out on the freemium example below. One metric there has a 14.2 percent minimum detectable effect. Another has 61.2 percent, at the same sample size, in the same business.

Hard to game

Do not think of gaming as fraud. Think of it as three specific adversaries who all behave rationally.

The spammer is external: fake accounts, scripted actions, bought installs. Ask whether 50,000 fake accounts appearing tomorrow makes your number rise or fall. A raw signup count rises. A signup count filtered to accounts completing a real action inside two days barely moves. And any engagement rate whose denominator is all accounts falls, which is exactly the alarm you want.

The tired team is internal and honest. They are behind on a quarterly target and will find the least effortful path to it. Ask: what is the cheapest way for a competent team to hit this number without changing a user's experience for the better? Usually the answer is a denominator trick or a definitional narrowing.

The partner is the third party whose interests only partly overlap with yours: hosts, sellers, drivers, creators, advertisers. Ask: does this metric reward them for the behavior I want, or for withdrawing? The response time example below is exactly this case, and the naive metric pays partners to stop responding.

Attributable to someone who can move it

A metric that only moves when the macro economy moves is context, not a target. Ask which team owns it and what lever they pull. If the answer needs four organizations to cooperate, it is a board-deck number. Decompose it into owned sub-metrics before any team is judged on it.

Stable enough to read on your cadence

A number whose week over week standard deviation is 9 percent cannot be reviewed weekly. Most of what leadership sees is noise, and they will react to it anyway. Widen the window, change the estimator, or say up front that this is a monthly metric.

Interview tip: When you propose a metric, immediately state its expected baseline and its noise band, for example "we expect about 70 percent, and week to week it wanders by roughly one point." Interviewers rarely hear this and it lands as operational experience.

---

Two datasets to argue with

Every claim below is computed on two tables, and both are described here so you can follow the arithmetic without guessing at the schema.

The first is Perch, a marketplace where people rent workshop and studio space. One row is a single booking inquiry sent to a host. It carries the inquiry id, the host's tier (new, established, or pro), and the region. It also carries whether the host replied at all, and how many hours the reply took. That last column is missing when no reply ever came, which turns out to matter enormously.

The second is Sandpiper, a free-to-play mobile game. One row is a single player-month. It carries the player id, a spend segment (free, minnow, dolphin, whale), the number of sessions that month, and revenue in that month. Most players spend nothing, so most of the revenue column is zero.

Perch carries 48,000 inquiry rows across five columns; Sandpiper carries 90,000 player-months across four.

Two orienting facts. On Perch, 78.1 percent of inquiries get any reply at all. Among those replies, the median wait is 2.4 hours and the mean is 5.2 hours. On Sandpiper, mean monthly revenue per player is 2.87 units of currency and the median is exactly zero.

---

Ratio or absolute: report both, always

Candidates pick one and defend it. The correct answer is that they answer different questions, and shipping only one is how a team fools itself.

A ratio measures quality per unit and is comparable across time, geography, and cohort size. Its weakness is that it is a weighted average, so it moves when the weights move even if the product did not.

An absolute count measures the size of the outcome the business banks. Its weakness is that it grows with traffic, so it flatters any team whose marketing is spending more.

The mix shift that makes a ratio lie

Perch's headline reliability metric is the share of inquiries answered inside 12 hours: 70.2 percent. Break it by host tier and the reason is visible.

The three tiers are wildly different. Pro hosts answer 91.5 percent of inquiries inside 12 hours; new hosts manage 48.7 percent. The headline 70.2 percent is nothing but those three numbers averaged with the share-of-inquiries column as weights. So anything that changes the mix changes the headline, without any host behaving differently.

Now run a supply push. Marketing onboards a wave of new hosts. The new tier's share of inquiries goes from 31.2 percent to 43.7 percent, and total inquiry volume rises 20 percent. Every single tier improves its within-12h rate by two full points. Here is what the two headline numbers do.

Every tier got better. The company answered 5,624 more inquiries inside 12 hours. And the headline ratio fell 1.9 points. If reliability is your only metric, this quarter reads as a regression and someone proposes slowing host acquisition, which is exactly backwards.

The reverse error is more dangerous because it looks like success: cut your weakest acquisition channel, watch every rate rise, and quietly ship fewer good outcomes.

That second row carries a condition worth saying out loud, because the supply push you just read about is a marketing spend change and the ratio lied about it. Spend is safe for a ratio when it buys more of the same population. It is not safe when it buys a different one, which is what a channel push usually does. Treat "the mix held still" as a claim to check, never an assumption to make.

The clean version of this in a review is a fixed-mix rate: recompute the current period's per-segment rates against last period's segment weights. Do that here and you get 72.2 percent against a starting 70.2 percent. That is the honest +2 points every tier actually earned. That single line separates "we got better" from "our population changed."

Interview tip: Whenever you name a rate, say the denominator out loud and then say what would happen to your metric if that denominator grew 30 percent. If you cannot answer instantly, you have not finished designing the metric.

---

Average, percentile, or share above a threshold

This is the highest-frequency version of the question, and it usually arrives as a forced choice. Perch's version: would you measure host responsiveness as the share of inquiries answered inside 12 hours? Or as the average reply time among replies that arrived inside 12 hours?

Both sound defensible. Only one is safe, and the reason is not statistical elegance, it is the incentive.

Why an average over responders pays partners to go silent

The average is computed only over hosts who replied. That means the population is chosen by the behavior you are trying to measure, which is a selection problem dressed as an aggregation choice. A host who never replies is simply absent from the denominator, so the worst possible outcome for a guest is invisible to the metric.

Suppose the hosts who currently reply slowly, between 3 and 12 hours, stop replying at all. No product change, no improvement for any guest, purely withdrawal. Both metrics react, in opposite directions.

The average improved from 2.95 hours to 1.28 hours. On a slide that is a 57 percent improvement in responsiveness, produced by destroying 12,198 replies. The threshold metric did the honest thing and fell 25 points, because non-response counts as failure rather than as absence.

This generalizes. Any average computed over the subset that performed the action is gameable by shrinking that subset. Support raises average satisfaction by not surveying angry customers. Sales raises average deal size by disqualifying small leads. The metric improves and the business does not.

The structural fix is to define the metric over the full eligible population and let failure enter the numerator as a zero. A rate does this naturally. An average does not.

The cost of a threshold: nothing happens inside the bucket

Threshold metrics are not free. A reply at 40 seconds and one at 11 hours 50 minutes score identically. So a team already at 70 percent has no incentive to make good experiences excellent, only to drag stragglers past the line. In marketplaces this leaves a visible artifact: replies piled up just under the deadline.

A second cost candidates miss: a single threshold is a cliff, so the metric is maximally sensitive to hosts sitting near it and blind everywhere else. Move it from 12 hours to 10 and the same product looks different.

The two-threshold pattern

The production answer to a forced choice between an average and a threshold is usually to refuse the choice. Ship a pair instead. One aggressive threshold represents excellence, and one generous threshold represents an acceptable floor. Then add a rule: neither may fall while the other rises.

On Perch, 19.6 percent of inquiries are answered within 1 hour and 70.2 percent within 12. Together they describe the distribution far better than any single statistic, and they close each other's loopholes. Bunching replies at 11 hours 59 to protect the 12-hour number does nothing for the 1-hour number. And cherry-picking easy inquiries to chase the 1-hour number makes the 12-hour number sag.

Bunching is the loophole worth understanding, because it is the one a generous threshold cannot see on its own. Drag every straggler to just under the line and the 12-hour share climbs toward its ceiling, which is the reply rate itself, 78.1 percent. The 1-hour share does not move a hair, because nothing that happens inside the 12-hour bucket touches it. That gap between the two numbers is the whole point of running both.

The withdrawal cheat, meanwhile, is already handled by the 12-hour threshold on its own. The table above showed it: silencing the slow repliers collapsed that number from 70.2 to 44.8, exactly as a well-built metric should.

tradeoff matrix

Choosing the aggregation for a skewed duration

Choosing the aggregation for a skewed duration

Aggregation	Strength	Weakness	Use when
Mean over actors who acted	Uses the full magnitude	Gameable by withdrawal, outlier driven	Almost never as a headline metric
Median over actors who acted	Robust to outliers	Still excludes non-actors, insensitive to tails	Diagnostics, never as a target
Share within threshold T	Includes non-actors, easy to explain	Flat inside the bucket, cliff at T	Default choice for reliability and latency
Pair of thresholds, fast and floor	Rewards excellence and coverage	Two numbers to explain	Any partner-facing service level
p90 or p95 of the full population	Tail focused, sensitive to worst cases	Noisier, needs non-actors imputed	Engineering latency, ops queues

Note the last row. Percentiles are computed over the full population only if you decide what a non-reply is worth. The clean convention is to impute non-response as the maximum of the window, for example 168 hours for a one-week measurement period. Then p90 is well defined, and a host who never replies is counted as the failure they are.

That convention carries a precondition nobody states, and Perch fails it. Imputed percentiles only work when non-response sits below 1 minus the percentile. Perch's non-reply rate is 21.9 percent, so once you impute 168, every percentile from p80 through p99 is exactly 168.000. The imputed non-repliers alone fill the top fifth of the sorted list. Make every real reply ten times faster and p90 does not move by a second.

The arithmetic boundary is p78.1, but the usable ceiling sits lower, and the reason is how a percentile is actually computed. It does not pick a value out of your data. It lands between two neighboring values in the sorted list and interpolates linearly between them. So as you climb toward the 78.1 boundary from below, each percentile is a blend of one real reply and one imputed 168. It drifts upward toward the ceiling. p75 is 21.8 hours, and it is a reply somebody actually sent. But p77.5 is 47.5 and p78 is 83.3, numbers that look real but are mostly the 168s bleeding downward. Treat p75 as the limit and fall back to a threshold rate above it, which is the reason the two-threshold pair beats a percentile here. Percentiles suit engineering latency and ops queues because non-completion there is a fraction of a percent, not a fifth of the population. One wrinkle: 9 Perch replies exceed 168 hours, the slowest at 265.9, so the window maximum is a censoring floor, not a true maximum. A censoring floor means a 168 in the data reads as "at least 168", not "exactly 168", so anything you compute off those rows understates the real wait rather than measuring it.

---

Per-user normalization and the denominator nobody wrote down

Three metrics that get called the same thing:

1. Total actions in the week.

2. Actions per active user in the week.

3. Share of users performing at least k actions in the week.

They disagree constantly and they answer different questions.

Total actions is a business volume number and drifts with the user base. Actions per active user is the classic per-user normalization, and its problem is that it is a mean, so a small heavy-usage cohort dominates it. Share of users above a threshold is the robust form. It is what you want whenever the top of the distribution is not where the money is.

That second row is the one that ends interviews. Engagement per active user rises when your least engaged users leave, so a product that is quietly bleeding casual users shows an improving engagement metric while it dies. Anytime you use an activity-conditioned denominator, you owe a paired absolute count. Ideally you also owe a fixed-cohort version, where the denominator is frozen at the start of the period.

The fourth row matters too. A session is not a natural object. It is whatever your inactivity timeout says it is, usually 30 minutes. So any metric with sessions in the denominator can be rewritten by an analytics config edit nobody announces.

Interview tip: If an interviewer offers you "per user" without saying which users, ask. "Per registered, per monthly active, or per user eligible for the feature" is a five-second question that reframes you as someone who has shipped a dashboard.

---

When you actually want the outliers: freemium and whales

Everything so far pushed toward robust statistics. Now the exception, and it is the exception interviewers use to find out whether you learned a rule or a reason.

Sandpiper gives the game away free and sells items inside it. Its revenue column, five ways:

The median is zero, and it will be zero next quarter and the quarter after, because 93.9 percent of players never spend anything.

Be precise about why the mean sits so far above it, because interviewers push here. More than half the players sit exactly on the floor, so the median is pinned at zero no matter what happens above it. You could triple every payer's spend and the median would not move. Meanwhile the paying 6.1 percent drags the mean up to 2.87, and 180 of them carry 64 percent of revenue on their own. That ordering comes from where the mass sits plus a heavy tail. It is not a fact about the range. "Non-negative with no upper bound" does not by itself put a mean above a median, and plenty of non-negative distributions run the other way. Here every robust statistic you were just taught to prefer is blind to the business.

The number that matters here is the last one. Roughly 180 players out of 90,000 produce 64 percent of revenue. Suppose your metric is the median, or a trimmed mean, or the share of players above a spending threshold set at any sane level. Then you have chosen a metric that cannot see two thirds of the company. That is a disqualification as a business metric, not as an experiment readout. Payer conversion is a share above a threshold, and it is the best-powered readout in this whole dataset. The next section explains why those are two different jobs. When someone asks whether mean or median revenue per player is larger, the answer is mean. Volunteer the follow-up too: this is precisely why revenue metrics for freemium products are averages and not percentiles.

What that costs you in detection power

Outlier sensitivity is not free. A metric that a handful of players can move is also a metric that a handful of players make noisy. The per-arm sample is 12,000 players, and the minimum detectable effect is quoted at 80 percent power and a 5 percent two-sided test.

One formula does all the work:

MDE = (zα/2 + zβ) × σ × √(2/n)

Every piece earns its place. σ is the standard deviation of the metric across users. σ × √(2/n) is the standard error of the difference between the two arms. That is the amount two identical arms of n users each would differ by, purely from which users landed where. zα/2 = 1.96 is the false-positive threshold. On a 5 percent two-sided test, a gap must exceed 1.96 standard errors before you are willing to call it real. zβ = 0.84 is the extra distance you need. It makes the true effect land reliably past that threshold, rather than falling just short of it half the time. That 0.84 buys you 80 percent power. They add because a real effect has to clear both hurdles. It must be far enough out to be declared significant. And it must be far enough out that it clears the bar 80 percent of the time rather than 50. The √2, finally, is the two-arm structure of the experiment. You are differencing two independent estimates of n users each, so the variance you face is 2σ²/n rather than σ²/n. Change the power, the alpha, or the split, and only the pieces you touched move.

Run that formula over four candidate readouts and the spread is the whole lesson. The absolute detectable difference is then divided by the metric's own mean, which turns it into the relative lift a product team actually quotes. That is the last column below.

One of those four candidates carries a rule with teeth: the mean capped at the 99th percentile. Fit the cap once, on pooled pre-experiment data, and freeze it. If you instead recompute the 99th percentile separately inside each arm, the cap becomes an outcome the treatment moved. A feature that creates more whales produces a higher cap in the treated arm, which promptly caps that feature's own effect away. You would have built a metric that suppresses exactly the wins you were hunting.

Note the parenthesis on the third row. It is computed on the 732 payers per arm rather than the 12,000 assigned players. It also sits on a distribution with roughly four times the standard deviation. So its standard error lands about sixteen times larger. Both factors contribute about equally; the sample size alone would only explain half of it.

That larger standard error does not make it a less sensitive metric, and this is the trap in reading the third column against the fourth. Its mean is also about sixteen times larger, so the detectable relative lift comes out at 59.6 percent against 61.2 percent for revenue per player. Essentially the same. Raw standard errors are not comparable across metrics measured on different scales; only the relative lift column is. Revenue per payer gets disqualified below for a causal reason, not a statistical one.

Read the first two rows together. At the same sample size, payer conversion detects a 14.2 percent relative change, while mean revenue per player needs 61.2 percent. That is a factor of more than four in required effect. Put another way, it is roughly nineteen times the sample for equal sensitivity.

That does not mean you target payer conversion. It means your experiment reads on payer conversion and your business reads on revenue, and you must say so rather than pretend one is the other. Most game teams land on a split. Revenue per player is the north star, reviewed monthly on the full population. Experiments are powered on the decomposition: payer conversion times purchases per payer times average purchase value.

That last sentence hides a trap. Only payer conversion and unconditional revenue per assigned player are valid arm contrasts. Purchases per payer and average purchase value condition on an outcome the treatment moves. So the arms hold different sets of payers, and the gap between them is not a causal effect. It is a real effect tangled with a composition shift.

The direction is the tell. A feature that converts marginal, low-spend players adds them to the treated payer pool, dragging revenue per payer down while revenue per assigned player rises. Read only the conditional factors and you kill a winner.

So use the decomposition to locate a change, and stop short of quoting a lift on a conditional factor as an effect. If you must say something rigorous about it, there are two named routes. Principal strata means restricting the comparison to the players who would have paid under either arm, so both sides hold the same people. That is a well-defined group. But you usually cannot tell who is in it, which is why the estimate comes out partial. Lee bounds accept that treatment added payers you cannot match. They trim the treated payer pool from each end by the excess conversion rate. What you get is a worst-case and a best-case number rather than a point estimate. Either is defensible. The one thing that is not defensible is quoting the raw conditional lift as if it were an effect.

Winsorizing looks smart and deletes the business

The obvious statistician's move is to winsorize. Replace every value above a chosen percentile with that percentile's value, here the 99th, and take the mean of what is left. It is stable, its detectable lift improves from 61 percent to 20 percent, and it is still wrong here.

The capped mean is 0.80 against a true mean of 2.87. The cap keeps 27.7 percent of the revenue. So the trade is roughly three times the sensitivity, bought by deleting 72 percent of the business from the metric. Any experiment that works by making whales spend more now reads as flat. The team learns that whale features do not work, which is the opposite of true.

Capping is right when the tail is measurement error or abuse: a scraper generating 40,000 pageviews, a duplicated order, a bot session. It is wrong when the tail is the customer. The test is one sentence: would you be delighted to have more observations like the ones you are capping? If yes, do not cap them.

Interview tip: Say "I would cap outliers only if the tail is instrumentation or abuse rather than revenue" and then name which one it is in this business. That distinction is the entire answer to the winsorizing question.

The modeling consequence, briefly

The same logic reaches model choice. A tree splits on rank order and predicts the leaf mean, so it compresses the extreme tail toward its neighbors. For predicting who will spend at all, trees are excellent. For predicting how much a spender spends, a regression tree systematically under-predicts whales, and its errors are largest exactly where the revenue is.

Three fixes are standard. Split the problem into a classifier for "will they pay" plus a magnitude model on payers only. Or fit the magnitude model on a log scale. That tames the skew beautifully, and then hands you a subtle bill. Averaging logs and exponentiating gives you the geometric mean, which is always at or below the arithmetic mean, and the gap widens as the spread widens. So on the most skewed data, where the log transform is most tempting, the understatement is worst, and finance wanted the arithmetic quantity. A smearing estimator, or Duan-style retransformation, corrects for this if you go that route: exponentiate the log-scale prediction, then multiply it by the smearing factor, which is the average of the exponentiated residuals from that same fit. The factor is estimated from your own residuals rather than assumed, which is why it holds up when the errors are not normal. Or third, use a loss built for this shape, such as gamma or Tweedie regression. Both are generalized linear models fitted through a log link, so the prediction already lands on the original scale and the retransformation problem never arises. They differ in the variance they assume: gamma lets variance grow with the square of the mean, which holds the coefficient of variation constant, while Tweedie lets it grow with the mean raised to a power between 1 and 2 and, unlike gamma, admits an exact point mass at zero. That last property is what a revenue column where 93.9 percent of rows are zero actually requires.

---

Choosing a north star and its guardrails

A north-star metric summarizes the product outcome the team intends to improve. It is useful when the definition connects to user value and the team can explain what it omits. Pair it with guardrails for quality, costs, retention, or other outcomes that could worsen while the headline metric rises.

The first bullet below leans on an identity. Over any period:

active users at the end = active users at the start + new users − churned users

That is true by construction, not by assumption, and it is why the derivation starts where it does. There are exactly two terms on the right you can push on. Any candidate north star that moves neither acquisition nor retention is not a growth metric, whatever else it may be.

concept flow

Deriving a north star and its guardrails

1. 1
   Start from the growth identity

   growth needs new users and retained users, so every candidate must plausibly move one of the two

2. 2
   Pick the moment of realized value

   the single user action that best predicts the user is still here in 90 days

3. 3
   Wrap it in a population and a window

   which users count, over what period, so two analysts compute the same number

4. 4
   Ask the three adversaries

   what does a spammer, a behind-schedule team, and a partner each do to hit this cheaply

5. 5
   Add one guardrail per cheat

   each guardrail must fall if the corresponding shortcut is taken

6. 6
   Add one quality guardrail and one cost guardrail

   user-reported quality and unit economics, regardless of the cheats

7. 7
   Set a review cadence and a noise band

   state the expected value, the week to week wobble, and the ship rule

Guardrails are not a wishlist of nice numbers. Each one exists because a specific shortcut exists. Three fictional products, worked through:

Two of those cells repay a second look.

Sandpiper's denominator is registered players, not active ones, and that is deliberate. It is the same unconditional quantity the power table above computed over all 12,000 assigned players, including the 93.9 percent who never pay. Switch it to per-active-player and you have built the exact failure this lesson called the one that ends interviews. Revenue per active player climbs on its own as bored free players stop logging in. The number improves while the game empties out.

Ledgerline's guardrail names a narrow cheat on purpose, because the north star is narrow. It counts distinct businesses that sent at least one invoice, so unsent drafts cannot pad it. A business already counted contributes exactly 1 whether it sends one invoice or fifty. The cheat that survives is a business with nothing real to bill, resending a stale invoice to stay in the count. That is the one the guardrail catches: an invoice sent to keep a streak alive does not get paid within terms.

Read the guardrail column carefully. Sandpiper's north star is revenue. The cheapest way to raise revenue this quarter is to make the free experience worse until more people pay to escape it. That works for two quarters and then the funnel dries up. The guardrail is retention of the people who are not paying, which is the population the cheat harms. That is the shape a guardrail should always have: the metric that the shortcut damages.

One more property worth stating aloud. As a company matures, it replaces the north star with a narrower proxy a team can move in a sprint. An example: the share of new hosts who upload three photos in week one. That substitution is legitimate only if you have shown the proxy predicts the north star and moves when the team's own levers move. Without that evidence you have handed a team a target unconnected to the business, and no product manager will fund work against it. That tension gets its own lesson later in this section.

---

Metrics for a team that does not own a product surface

"How would you measure customer support" is the same question with the surface removed. It separates candidates faster than almost anything else, because there is no obvious event to count.

The weak answer lists operational numbers: average handle time, tickets per agent, first response time, satisfaction. Each is gameable in an afternoon. Handle time falls if agents close tickets fast and unhelpfully. Tickets per agent rises if you split one issue into three. Satisfaction rises if you only survey resolved tickets.

The strong answer starts one level up. Support exists to protect growth, and growth comes from acquiring users and keeping them. Acquiring a customer costs far more than retaining one. That is why companies hand out credits so freely after a bad experience: the credit is cheap next to the replacement. So the target for support is the incremental lifetime value it preserves, and every operational number is a candidate proxy for that, not a substitute.

But lifetime value takes a year to observe, and you cannot run an organization on a metric with a one-year lag. So you make the standard long-horizon move. Find a short-run signal that predicts the long-run outcome and also carries the effect of the changes support can actually make. Then manage on that signal. Prediction alone is not enough.

The pull, on a support ticket table joined to orders:

WITH t AS (
  SELECT ticket_id,
         account_id,
         opened_at,
         first_reply_minutes,
         resolution_minutes,
         reopen_count,
         csat_score,
         sentiment_delta,
         issue_topic
  FROM support_ticket
  WHERE opened_at >= DATE '2024-01-01'
    AND opened_at <  DATE '2025-01-01'
)
SELECT t.*,
       MAX(CASE WHEN o.placed_at >  t.opened_at
                 AND o.placed_at <= t.opened_at + INTERVAL '365 days'
                THEN 1 ELSE 0 END) AS bought_within_year
FROM t
LEFT JOIN orders o
  ON o.account_id = t.account_id
GROUP BY t.ticket_id, t.account_id, t.opened_at, t.first_reply_minutes,
         t.resolution_minutes, t.reopen_count, t.csat_score,
         t.sentiment_delta, t.issue_topic;

Then fit a model of bought_within_year on the ticket features. Include text-derived ones too, such as topic and the change in sentiment between the customer's first and last message. Two outcomes are possible and you should name both.

If one variable carries most of the signal, make that the team's metric. Post-contact satisfaction often wins. Before adopting it, walk the levers support holds (speed, training, credits, deflection, root-cause fixes) and ask of each whether it moves that metric. Deflection and root-cause fixes are the awkward pair. They raise one-year purchase by removing the ticket entirely, so they produce no post-contact score and read as null. Those are the changes you most need to see. The coefficient says the variable predicts the outcome; it is not evidence your lever works.

If no single variable is enough, use the model's predicted probability as the metric. That is a legitimate answer and worth saying, because it shows you are comfortable with a composite target. But volunteer the three ways it goes wrong, because the interviewer will otherwise ask. Only the first is specific to a composite. The other two hit the single-variable pick just as hard, since both come from the same observational fit.

• The composite is gameable through its inputs. Once agents know sentiment recovery drives the score, sentiment recovery gets engineered. Rotate the feature audit and hold out a raw outcome you never expose.

• The relationship drifts. A model fit on last year's ticket mix decays as the product changes. Refit quarterly and monitor whether the historical link between the proxy and the one-year outcome still holds.

• The population is selected. Only customers who contacted support are in the sample, and they differ from everyone else. The model tells you which contacts go well, not whether contact itself helps. Answering that needs a comparison against similar customers who had the same problem and did not write in. That is a matching or experiment question, not a modeling one.

Interview tip: For any team without a product surface, say the ladder out loud: growth, then retention, then lifetime value, then a short-run proxy that predicts lifetime value, carries the effect of the levers you hold, and has been checked against past cases where the truth eventually arrived.

---

Write the spec down

A metric that lives in someone's head gets redefined every quarter and the series becomes uninterpretable. Saying you would write a one-page spec is a small thing that reads as real experience.

checklist

The metric spec, one page

• Name and one-line intent: what decision this number informs

• Population: exactly which entities are in the denominator, and the exclusions

• Numerator condition: the event, the threshold, and the attribution rule

• Window and cadence: measurement period, reporting frequency, and any lag before the number is final

• Current baseline and noise band: expected value and the typical period to period wobble

• Owner: the single team accountable, and the levers they hold

• Guardrails: each one paired with the shortcut it exists to catch

• Known gaming vectors: written down deliberately, including the ones you decided to accept

• Change log: every redefinition, dated, with the old and new series overlapped for one period

That last item prevents the most expensive metric failure there is: an unlogged definition change. You discover it six months later, when someone tries to explain a step in the series.

---

Common traps

Proposing a metric with no population or window. "Engagement" and "retention" are directions. Fix: never say a metric name without saying who is counted and over what period.

Averaging over the people who acted. Any mean computed on completers is improved by removing the marginal completer. Fix: define over the full eligible population and count failure as a zero.

Reporting a rate without its count. A rate rises when weak users leave and falls when you grow into a harder segment. Fix: pair every rate with an absolute, and add a fixed-mix version when the population is moving.

Applying robustness reflexively. Medians, trims, and caps are correct for skew caused by noise and wrong when the tail is the revenue. Fix: ask whether you would want more observations like the ones you are discarding.

Choosing a metric you cannot detect a change in. A north star with a 60 percent minimum detectable effect will read flat for every experiment you ever run. Fix: compute the standard error at your realistic sample size before you commit. If it is hopeless, power on a decomposed factor that is still defined over the full assigned population, and say so.

Shipping a north star with no guardrails. Every optimizable number has a cheap path. Fix: one guardrail per named shortcut, each one being the metric that shortcut damages.

Using an activity-conditioned denominator without a companion. Per-active-user metrics improve automatically as casual users churn. Fix: report the absolute alongside, or fix the cohort at period start.

Adopting a narrow proxy without proof. The proxy must predict the broad metric, and it must carry your levers' effect on that metric. Profile photo completion is not a growth metric until you have shown it moves growth. It is still not one if your shippable changes bypass it. Fix: state both pieces of evidence, or that establishing them is step one.

Letting a session-based denominator go unexamined. Sessions are defined by a timeout setting, so a config change silently rewrites your history. Fix: prefer user-based denominators for anything on a leadership dashboard.

Redefining mid-answer under pressure. If an interviewer pushes and you swap definitions, everything you computed becomes invalid. Fix: defend the original, or explicitly retract it and restate what changes.

---

Quick self-check