1.1 How to Improve a Given Metric

A product data science interview may begin with a broad question such as "How would you improve engagement?" A useful answer explains how a proposed change could improve a defined outcome. This lesson develops a seven-step approach: turn the goal into a computable metric, identify where the metric loses value, and rank changes by their expected contribution. The result is a test plan that explains what to build next and how to judge whether it worked.

Why this matters in interviews

An open-ended product prompt can tempt you to start with features: better notifications, referrals, or a redesigned home feed. First explain the problem and the evidence that makes a particular intervention relevant. That reasoning lets someone assess the proposal rather than only its novelty.

Use four questions to check the answer as it develops. Is the goal defined well enough to compute? Can the resulting number be decomposed into parts? Which part accounts for the largest loss after accounting for volume? What change addresses that loss, and how would you measure its effect? These questions make the reasoning visible and help you notice a missing step before choosing a feature.

Notice this is an improve question, not a diagnose question. Nothing broke; the metric is simply below where someone wants it. Diagnosing a sudden movement is a different discipline with different first moves, and it gets its own lesson later in this section. Open an improvement question by asking about last Tuesday's logging change and you have answered a question nobody asked.

Interview tip: Say out loud in your first thirty seconds that you are going to define the metric, decompose it, localize the loss, separate levers from context, generate hypotheses, rank them, and propose a test. Naming the plan buys you the room to execute it.

The rest of this lesson runs that plan on a concrete product. Then it does the marketplace version, where the loss is not in one funnel but split across two sides that want different things.

---

Step 0: turn the goal into a metric you can compute

Engagement is not a metric. Neither is retention, growth, quality, or satisfaction. Those are directions. A metric is a number with a numerator, a denominator, a population, and a window, such that two people with the same data compute the same value.

The test a definition has to pass

Before you accept your own definition, run it through four checks.

Can you write the query. If you cannot describe the table, the filter, and the aggregation, the definition is a slogan. "Users are more active" fails. "Share of accounts active in the last 28 days that log at least one cooked meal in a calendar week" passes, because you can see the SQL.

Does it move only when the product gets better. Page views per session sounds like engagement. It is famously satisfied by making things harder to find.

Is it aligned with what the company is trying to be. Candidates skip this, and it is the part that reads as senior. If your metric can rise while the company's reason for existing falls, you chose wrong.

Is it stable enough to read. A metric too noisy week to week to distinguish a five percent change from nothing is unusable as a target, however pure. Sensitivity and stability get a full treatment in the next lesson.

Weak answer, stronger answer

Four fictional products, each moving from a vague goal to something computable.

Every entry in the last column has a population, a numerator condition, and a window. Each one closes the cheat available in the weak column. Almanac's cannot be gamed by dumping low-effort answers, because the upvote threshold makes volume without quality worthless. Halyard's cannot be gamed by the sales team handing out seats. If the interviewer disagrees with your definition, that is a productive disagreement. Having no definition, and improvising a new one every two minutes, is not recoverable.

For the rest of this lesson the running example is Kettle. The metric is the one in the table: weekly cook rate, the share of active accounts that log at least one cooked meal in the week.

---

A dataset to make this concrete

Everything below is computed on a single account-week table for Kettle. One row is one account in one week.

The four flags nest. An account cannot save a recipe in a week it never opened the app, and cannot log a cook in a week it never saved a recipe. That nesting is what lets the funnel in Step 1 multiply cleanly, with no residual to argue about.

The table is 60,000 rows by 9 columns, and the weekly cook rate across it is 11.18 percent. That is 6,711 accounts out of 60,000. Hold onto that denominator: every proposal later is sized against it.

---

Step 1: write the metric as arithmetic

Two decompositions are worth knowing, and a strong answer uses both. They answer different questions and suggest different kinds of fix.

The multiplicative chain

A rate that requires several things to happen in order is a product of conditional rates. For Kettle:

weekly cook rate = P(open) x P(start a plan | open) x P(save a recipe | start a plan) x P(log a cook | save a recipe)

Each factor is a step where people leave. Multiplying them reconstructs the metric exactly, and that exactness is what makes the decomposition trustworthy.

The plan-start step has both the worst pass rate, 40.6 percent, and the largest absolute loss, 23,578 accounts. Those two facts do not always coincide. When they disagree, follow the volume, but convert each loss into metric units first, because losses at different depths of the funnel are not the same thing.

An account you rescue at the open step still has to survive plan start, recipe save, and cook logging before it becomes a cook. That is 0.4062 x 0.7235 x 0.5751, or about 0.169 cooks for each account recovered. An account you rescue at plan start only has to survive two steps, 0.7235 x 0.5751, or about 0.416 cooks each. So the 20,291 lost at open are worth about 3,429 recoverable cooks, and the 23,578 lost at plan start are worth about 9,810. The ranking survives here, but it would not always: a larger raw loss at a shallow step can be worth less than a smaller loss deep in the funnel.

A step with a 20 percent pass rate that only 300 people reach is not worth a quarter of engineering time, whatever its rate looks like.

The additive mix

The second decomposition slices the same metric across a dimension. If accounts fall into disjoint segments with weights that sum to one:

metric = sum over segments of (segment share) x (segment rate)

The decomposition distinguishes changes in segment rates from changes in population mix. Product changes can affect either, as can acquisition, eligibility, and targeting. Inspect both the rate and total outcome count before deciding whether a higher aggregate rate represents an improvement.

Sliced by tenure, the same 11.18 percent breaks apart like this.

The cooks-left-on-the-table column does the real work. If this segment behaved like the best comparable one, how many additional accounts would log a cook. That converts a rate gap into volume, and volume is the unit you need to compare one proposal against another.

Why you need both

The chain tells you where in the experience the loss happens. The mix tells you who it happens to.

A step-only answer redesigns the plan-start screen for everyone and gets a diluted effect, because two thirds of accounts were already fine there. A segment-only answer says "new users are weak" without naming a screen. The move that separates a strong candidate is crossing them: pass rates per step, computed inside each segment.

concept flow

The seven-step answer

1. 1
   Define the metric

   numerator, denominator, population, window, and why it matches the mission

2. 2
   Decompose

   multiplicative chain for where, additive mix for who

3. 3
   Localize

   rank by accounts lost, not by rate gap

4. 4
   Separate levers from context

   keep the segments you can act on, drop the ones you can only describe

5. 5
   Hypothesize

   one testable cause per weak segment-step cell

6. 6
   Rank

   addressable volume times plausible lift times confidence, divided by build cost

7. 7
   Propose

   unit of randomization, primary metric, guardrails, minimum detectable effect, run length

---

Step 2: find where the loss is concentrated

Now cross the two decompositions. Here is the conditional pass rate at every step, computed inside each segment.

By tenure

By platform

By country

By acquisition channel

Reading the grid

Read it as a heat map and hunt for isolated weakness: one bad cell in an otherwise normal row.

The rule gets asserted more often than it gets argued, and the mechanism behind it is simple. A defect in the product sits at one place in the sequence: a button, a screen, a search box. It can only remove the people who are standing at that step when they hit it. The same accounts sail through every other step at ordinary rates, because nothing they touch there is broken. That is what an isolated weak cell is: a surface failing, with normal behaviour on either side of it.

A different mix of people works the other way. Someone less likely to cook carries that disposition into every step they touch. Their weakness shows up as a broad tilt across the row rather than a single dark cell. Uniform weakness is usually a population difference, and it is much harder to act on, because you cannot ship a fix for who someone is.

One exception matters and the lesson leans on it later: a confounding variable that only bites at one step will imitate a mechanism perfectly. Step 2 ends with the check for that.

Three cells stand out.

India loses at plan start and nowhere else. Indian accounts open at 66.5 percent, the same as everyone. They save and cook normally once in a plan. But only 28.2 percent of Indian openers start a plan, against 47.0 percent in the United States. That points at the plan-start experience specifically: catalogue coverage, ingredient availability, unit systems, language, or a default meal structure built for the wrong cuisine.

Web loses at recipe save. Web opens less, as you would expect from a surface people reach deliberately. The interesting cell is save-given-plan at 55.8 percent against about 74 percent on both mobile platforms, with plan-start identical across all three. Same plan rate, then failure to save. That smells like an interface defect, not a preference.

Paid social loses at the very last step. Every upstream rate matches the other channels almost exactly. Then cook-given-save collapses to 49.4 percent against 60.6 percent organic. These accounts complete the whole flow and do not cook. That is intent, not product, and the fix lives in targeting or in creative that sets a different expectation.

Interview tip: When you find a weak cell, say what the rest of the row looks like before you propose anything. "India is weak only at plan start, everything downstream is normal" is a diagnosis. "India is weak" is an observation.

One more pattern deserves naming, because candidates over-index on it. Week-one accounts are weak at plan start (23.5 percent) and at cook logging (37.0 percent). In between, at recipe save, they are entirely normal at 72.3 percent.

That is what "separated" means: two weak steps with a healthy step sitting between them. It is the signature of a different population rather than a broken surface, and the argument is a cost-of-explanation argument. One product defect would have to live in two different places in the flow and skip the step in between, which takes two coincidences. One kind of person carries the same disposition to both steps, which takes none. New accounts include people who signed up out of curiosity and were never going to cook. Onboarding can recover some of that, but expect a lower ceiling than the raw gap suggests, and say so.

What would overturn it: a shared surface. Suppose the same recommendation widget appears on both the plan-start screen and the cook-logging screen. Then one defect explains both weak cells, and the population story loses its advantage.

There is one thing in the tenure grid you should raise before the interviewer does. Tenure is not monotone at the open step. Month-one accounts open at 55.8 percent, worse than week-one's 62.9 percent, which is not what a simple "new users are weaker" story predicts. The week-one story survives it, because the two cells it rests on, plan start and cook logging, do order cleanly (0.2351, 0.3705, 0.4597 and 0.3702, 0.4923, 0.6214). But the open-step reversal is real and unexplained, and naming it costs you nothing while pretending not to see it costs you a lot.

One check before any of this becomes a proposal

Every grid above is a marginal slice, so a weak cell can be an echo of a dimension you did not slice on. If Indian accounts skewed web, or week-one, the country table would look exactly like this while the mechanism sat somewhere else, and eight engineer-weeks of localization would buy nothing.

The isolated-weakness rule does not protect you here. A confounder that reaches only one step produces exactly one weak cell and reads as a mechanism. So recompute the weak rate inside the other dimensions, against the same reference group, before you propose anything.

Tenure mix within each country

Plan start given open, by country and tenure

India's tenure mix sits within a percentage point of every other country, and its plan-start rate is weak inside all three tenure levels against the same United States reference. The same pair of tables computed across platform instead of tenure, not shown here, says the same. Mix sits within a percentage point, and India comes in at 0.2772, 0.2894 and 0.2774 on android, ios and web against 0.4629, 0.4748 and 0.4827. The gap is not a mix artifact and the proposal survives. This is the first follow-up an interviewer asks about the India finding.

Know what the failing case looks like, because you just saw the passing one. A check that only ever passes teaches you nothing about how to read it. So picture the other outcome on these same two tables.

Suppose Indian accounts were mostly on web while every other country sat near the platform average. The country grid would look identical to the one above: India weak at plan start, normal everywhere else. A marginal slice cannot see the platform mix hiding inside it. The mix table is where it surfaces: India's web share would sit far above every other country's instead of within a percentage point. And the second table would then flatten. India's plan-start rate inside each platform would come back close to the United States rate inside that same platform. The country label was never doing the work. The platform was.

Two tables, two signatures. Lopsided mix plus flat within-cell rates means you found a confounder and the localization proposal dies. Flat mix plus a gap that survives inside every cell (which is what happened here) means the segment is real. Say which one you got.

---

Step 3: separate levers from context

Not every segment you can slice is one you can act on. This is where otherwise good answers go wrong: the candidate finds a real gap, then proposes something the company cannot do.

Two rows need a comment. Tenure is context by construction, but it is still a useful frame. You cannot make an account tenured. What you can change is what happens during week one, so fewer accounts stall there. Country is the interesting hybrid. You cannot change India, and you can change the catalogue, the units, the default template, and the language. The India cell is worth a proposal and the tenure cell is worth less, precisely because the India weakness sits at a step you own.

tradeoff matrix

Which kind of lever to reach for

Which kind of lever to reach for

Lever type	Strength	Weakness	Use when
Change the experience in a weak segment	Directly attacks the mechanism, testable in weeks	Only pays on that segment's volume	The weakness is isolated at one step you own
Shift acquisition mix toward strong segments	Raises the total without touching the product	Slow, capped by channel supply, can raise cost per acquisition	A high-rate segment is small and channel supply exists
Remove a step from the chain	Multiplies through everything downstream	Often a large build, may hurt a guardrail	A step exists for legacy reasons and adds no value
Change the metric's own definition	Free	Changes nothing real and will be seen through	Never, in an interview

The last row is not a joke. Candidates propose it accidentally, by drifting to a looser definition partway through so the number looks better.

---

Step 4: generate hypotheses per segment

You now have three or four localized weaknesses. Each needs a hypothesis specific enough to be wrong.

The form to use: for [segment], at [step], [rate] is [current] versus [reference], because [mechanism], so [change] should move it to about [target]. If you cannot fill in the mechanism, you do not have a hypothesis. You have an observation with a feature attached.

Bad segments are product work

Take the India cell. Candidate mechanisms, from most to least testable:

1. The default plan template assumes a Western week of dinners, so the first screen shows food people do not cook, and they abandon before committing.

2. The catalogue has thin coverage for the ingredients people actually buy, so search inside plan creation returns little.

3. Measurement units and portion defaults are wrong, making every recipe feel like extra work.

4. Payment or grocery-partner integration is absent, so a downstream promise in the plan-start flow is empty.

Each implies a different change and a different measurable intermediate. Hypothesis 1 predicts abandonment on the template picker specifically. Hypothesis 2 predicts a high zero-result search rate inside plan creation. Both are checkable before you build anything, which is what you should say you would do.

Good segments are acquisition work

The same mix decomposition run across acquisition channel puts the numbers behind the next argument on the page.

Search and organic accounts cook at about 11.8 percent, paid social at 9.7 percent. Paid social is 23.5 percent of the base, and the parity gap is worth 303 cooks a week. Two distinct moves are available, and they are not the same proposal.

The product move fixes what paid social accounts hit at the last step. They complete every earlier step normally, which argues against a broken surface.

The mix move reallocates budget toward channels whose users cook, or changes the creative so it selects for people who intend to cook rather than people who like the video. Moving weight from a 9.7 percent segment to an 11.8 percent segment raises the blended metric without changing anyone's behaviour.

Acquisition segments are not exchangeable. Cutting paid social does not imply that the same accounts arrive through search instead. Total cooks can fall while the rate rises. A mix change may still be justified by acquisition costs or contribution, but the decision needs those quantities; a higher rate alone does not establish success.

Interview tip: Whenever you propose a mix shift, immediately state what happens to the absolute count, not just the rate. Interviewers plant this trap on purpose.

Where hypotheses come from

Three sources, in the order you should reach for them.

First, the funnel grid: isolated weakness plus knowledge of what that step does.

Second, session-level behaviour in the fifteen seconds before the drop: whether people searched and got nothing, toggled a filter, or hit the bottom of a short list. This is where the model-reading techniques from section four earn their place. Fit a shallow decision tree. The target is the step outcome (did this account start a plan, one row per account), and the features are session behaviour plus segment attributes. Then read each leaf's path down the tree as a conjunction of conditions: web AND at least one zero-result search AND fewer than five results shown. That conjunction is a candidate segment definition, and it is one you would not have thought to slice on by hand. You would have sliced one dimension at a time. A rule set, a flat list of such conditions learned instead of written by hand, does the same job without the tree structure.

Two cautions on that. The tree is hypothesis generation, not causal evidence: it finds conditions that predict the drop, which includes consequences of the drop as easily as causes. And every leaf it hands you is a marginal slice like any other, so it still has to survive the mix check from Step 2 before it becomes a proposal.

Third, qualitative evidence: support tickets, reviews filtered to the segment, five interviews. Naming that last one is not soft. It shows you know a funnel tells you where and almost never why.

---

Step 5: rank by size times feasibility

You now have more ideas than quarters. Ranking turns the answer into a plan.

The score to use out loud: expected value = addressable volume x plausible relative lift x probability it works, divided by build cost in engineer-weeks.

Every term matters. Volume alone favours anything touching the biggest segment. Lift alone favours a heroic change to a tiny cell. Probability keeps you honest where the mechanism is a guess. Dividing by cost turns a wish list into a sequence. Size the volume term the same way for every candidate, so the numbers are comparable.

The downstream factor separates careful sizing from sloppy. Fixing plan start in India does not create 1,113 extra cooks, which is roughly the 9,448 openers times the 11.8-point gain from 28.2 to 40.0 percent, because only about 43 percent of the extra plan starters survive the two remaining steps.

Where the target rates come from. The four target rates are the single most consequential input in this whole section, so do not let them arrive unexplained. The rule is two moves.

Take the reference cell's rate as the ceiling. For India that is the United States plan-start rate of 0.4698. For web it is the mobile save rate of about 0.74. For paid social it is organic's cook-given-save of 0.6063. For week-one it is month-one's 0.3705, which is the nearest comparable rather than the best cell in the column.

Then discount the ceiling by the share of the gap your mechanism plausibly explains, and say the discount out loud. Take India. Its target of 0.40 recovers a little under two thirds of the 0.1876 gap to the United States. The argument: localization fixes template, catalogue and units, but not grocery infrastructure. Web's 0.70 recovers most of its gap, because an interface defect that you fix is fixed. Paid social's 0.56 recovers about six tenths, because targeting shifts intent partially and slowly. Week-one's 0.30 recovers just under half, because part of that gap is people who were never going to cook.

Those discounts are judgement, not measurement, and the ranking is extremely sensitive to them. Set India's target to the full 0.4698 benchmark and its extra cooks go from 473 to roughly 750. That is an 11.2 percent lift, which scores 0.84 and moves India from third place to second. One unstated assumption reorders the plan. Say the number and the reasoning behind it in the same breath, so the interviewer can push on the reasoning instead of guessing at it.

Now the ranking table. Total weekly cooks is 6,711, so the relative column is what matters to a reviewer.

Score is the lift column in percent, times confidence read off its own label with High 0.9, Medium 0.6 and Low 0.3, divided by engineer-weeks. India is 7.0 x 0.6 / 8 = 0.53. The web fix is 2.0 x 0.9 / 1 = 1.80. Onboarding is 1.7 x 0.3 / 6 = 0.085, shown as 0.09. Compute it from the columns as printed, so a reader can check every row without rerunning anything. If you prefer different units, scale every row by the same constant rather than picking one per row. Only three factors appear because the lift column already carries volume times plausible lift, being extra cooks over the 6,711 base. So all four terms of the Step 5 formula are present, with two collapsed into one column.

How to assign confidence, and what the engineer-weeks are. Confidence is the term that keeps you honest where the mechanism is a guess. So give it a rubric instead of a feeling.

High means the mechanism is directly observable in your instrumentation. The web save cell qualifies. Plan-start is identical across all three platforms, then save drops eighteen points on the one surface with its own front-end code path. You can go and watch the save interaction's event stream on web and see the failure.

Medium means the mechanism is one of several plausible stories that are all consistent with the grid, and you have not yet separated them. India qualifies: four candidate mechanisms were listed above and nothing in the data chooses between them.

Low means the weakness spans steps, or the whole thing could be a population difference rather than anything you built. Week-one onboarding qualifies on both counts.

The 0.9 / 0.6 / 0.3 weights are a convention, not a measurement. Their only job is to stop a confident-sounding guess outranking a near-certainty. Any mapping that decreases across the three labels and is applied to every row does the same work.

The engineer-week figures are placeholders, and you should say so. A candidate does not know Kettle's codebase, so 8, 2, 1 and 6 are numbers you would ask the interviewer for. The column's purpose is to force that question, not to be precise. So say the honest version out loud: "I would need engineering estimates here. Assume roughly one week for the web fix and roughly two months for the India work. On those numbers the order comes out like this." That reads far better than a confident 8.

The ordering by score is not the ordering by size. By volume the order is India, paid social, web, onboarding. By score it is web, paid social, India, onboarding. Only India and the web fix actually swap, and that swap is the point. The web fix is third by volume and first by score, because it is cheap and the mechanism is nearly certain. The India work is the biggest prize and still worth doing, just not first. Onboarding, which most candidates propose first, ranks last: expensive, and weak across two separated steps, which suggests population rather than mechanism.

Interview tip: Ship the cheap high-confidence fix first even when it is small, then use the time it buys to reduce uncertainty on the big expensive one. Saying that sequencing out loud reads as someone who has actually shipped.

---

Step 6: propose the test

A proposal without a test is an opinion. You do not need a full experimental design here, that arrives later in this course, but you do need six lines.

checklist

The six lines every proposal needs

• Unit of randomization: account, and say why not session or device

• Population: only accounts eligible to see the change, so the effect is not diluted by everyone else

• Primary metric: weekly cook rate, decided before the test, one metric

• Guardrails: recipe saves, unsubscribe rate, support contacts, and 7-day return rate

• Minimum detectable effect: chosen from what would justify the build, then checked against the sample the eligible population actually supplies

• Run length: enough weeks to cover a full behavioural cycle plus the novelty window

Three of those six lines use a term of art as if everyone already knows it. Define each one before you use it.

Unit of randomization is the thing you flip the coin over. Every row in your analysis has to belong to exactly one arm through that unit, and the unit determines what "independent observation" means.

Session randomization would flip a coin every time someone opens the app. That fails twice here. The same person sees both variants, so any carryover from the treatment session contaminates the control session and shrinks the contrast you are trying to measure. And sessions from one person are correlated, so treating them as independent understates the variance and inflates your significance.

Device randomization fails on a product people reach from a phone and a laptop: one person lands in both arms, and the same contamination follows.

Account is correct here because the outcome you are measuring, weekly cook rate, is defined per account per week. Randomize at the level your metric is computed at, and the two line up.

A guardrail measures an outcome that the change must not harm beyond an agreed tolerance. It helps detect cases where the primary metric improves while another part of the experience worsens. If the primary metric is recipe saves, repeated prompts could increase saves without increasing cooking; cook-given-save can help investigate that failure. Support contacts and 7-day return rate cover other possible harms. Define the acceptable deterioration and the response before the test. A small movement is not automatically a failure, but crossing the agreed threshold calls for investigation before rollout. Microsoft's discussion of metric interpretation explains this distinction between a guardrail and a metric that must remain completely unchanged.

A novelty effect is a response to a change being unfamiliar. Plotting the effect by time since exposure can reveal a transient response, but decay alone does not identify novelty: learning, seasonality, changing users, and measurement problems can produce similar patterns. Define a follow-up window appropriate to the product, and examine cohort composition and exposure history before extrapolating the initial lift.

Two points then distinguish a candidate who has run tests from one who has read about them.

Restrict the population to the eligible. Randomize all 60,000 accounts when only 7,784 can see a web change and you have diluted a real effect roughly eightfold. Randomize among the eligible, or analyse an eligible subset defined by pre-treatment attributes only, never by post-treatment behaviour.

That last clause is the most important causal rule in this lesson. Randomization gives you two groups that are comparable in every respect, observed and unobserved. Filter both arms on something the treatment could have changed, and that comparability is gone: you are no longer comparing randomized groups, you are comparing two self-selected subsets.

Concretely. Suppose the change makes marginal users start a plan who otherwise would not have. The treatment arm's pool of "plan starters" now contains people with lower intent than the control arm's pool, because control never recruited them. Measure save-given-plan-start and the treatment arm looks worse, even if the feature helped every single person. The difference you measured is the treatment effect plus the selection, mixed together with no way to separate them.

The test to apply is one question: could the treatment have moved this variable? Platform, country, signup date and acquisition channel are all fixed before exposure, so they are safe. Anything measured after exposure is guilty until you argue otherwise, and argue it explicitly rather than assuming it.

Pick the minimum detectable effect from the business, then check the sample can reach it. The sizing table already named the effect you are buying. Moving web save-given-plan from 55.8 percent to 70 percent is a 25 percent relative lift. And 133.7 extra cooks against the segment's 526 is the same 25 percent read on the outcome. Size for that. Choosing a detectable effect because it fits a week of traffic is how teams run tests that could never have answered anything. Choosing a small round one your eligible population can never reach is the same mistake pointed the other way.

Sizing the web test

Do the arithmetic out loud; the assertion is what gets audited. A handful of terms carry the whole calculation, so take them one at a time.

The significance level, usually written alpha, is the false-positive rate you are willing to accept: the chance you declare an effect when the truth is that there is none. The convention is 5 percent.

Two-sided means you would be interested in a move in either direction, so you split that 5 percent. That is 2.5 percent for "treatment is better" and 2.5 percent for "treatment is worse". The decision rule that comes out is: reject the null when the observed difference between arms exceeds 1.96 standard errors. That 1.96 is the normal quantile at 0.975, the point on a standard normal curve with 2.5 percent of the area above it. This is why the quantile is taken at 0.975 and not 0.95.

Power is the other kind of error. Given that a real effect of some stated size exists, power is the probability your test actually clears that 1.96 bar and finds it. Failing to find a real effect is a type II error, and power is one minus that error rate. The convention is 80 percent power: you accept a one-in-five chance of missing an effect you went looking for.

Now the two quantiles that get added together. For the test to reject 80 percent of the time, the true effect cannot merely sit at the 1.96 threshold. Park it exactly there and you clear the bar only half the time, because your observed estimate lands above or below the truth with equal odds. The effect has to sit far enough above the threshold that its own sampling noise pulls it back under only 20 percent of the time. That extra distance is 0.84 standard errors, the normal quantile at 0.80. So the effect must be 1.96 + 0.84 = 2.80 standard errors away from zero. That sum is the 2.80 in the table below.

The rule that falls out: the minimum detectable effect is 2.80 standard errors. MDE = 2.80 x SE in absolute terms. Divide that by the baseline rate to state it as a relative lift, which is how everyone quotes it.

Last, the standard error itself. A proportion p measured on n accounts has variance p(1 - p) / n. You are comparing two independent arms, and variances of independent quantities add, so the difference between two equal-sized arms sharing a rate p has variance 2p(1 - p) / n. The factor of 2 is the second arm, and it depends on the arms being the same size. Using one shared p for both arms is an approximation that says the treatment barely moves the rate. It is conservative to neutral when effects are small. It does start to strain at the 25 percent lift this test is chasing, and that is worth saying out loud rather than hiding.

Now the sample. You assign each account to one arm once, then observe it for two weeks. That is not the same as having twice as many accounts, and the difference matters.

The 7,784 eligible web accounts split into about 3,892 per arm. Running two weeks does not create new accounts; it collects a second observation on each of the same ones. What the second week buys is less noise per account. Averaging an account's two weeks gives a quantity whose variance is the single-week variance times (1 + rho) / 2. Here rho is the correlation between the same account's two weeks.

Both ends of that are worth knowing. Say rho is 0, meaning each week is an independent coin flip for that account. The average then has half the variance, so two weeks of account-weeks behave exactly like twice as many accounts, and n per arm is 7,784. Say rho is 1 instead, meaning an account either cooks every week or never does. The second week tells you nothing new, and n per arm is 3,892. Reality is in between, so plan against both.

The two candidate primary metrics size out like this.

Those figures are the optimistic, rho = 0 end. Weekly cook rate detects about a 17 percent relative lift there, which the 25 percent effect clears with margin. At the pessimistic rho = 1 end the same metric detects 23.6 percent, since halving the arm widens the standard error by the square root of 2, which turns 16.7 into 23.6, and the 25 percent effect scrapes past instead of clearing. Plan against 23.6 percent and check the account-level week-to-week correlation on a pre-period before you commit; that measurement tells you which end you are actually at.

Read the +3 percent power figure carefully, because it is easy to misread as the sizing operation. It is not. It answers a different question. If the true effect were only 3 percent, how often would this fixed sample find it? The answer is 7 percent of the time, barely above the 5 percent false-positive rate. That is a test that rejects almost as often when nothing is happening as when something is. Power collapses toward that floor as the effect shrinks. Going the other way and sizing for a 3 percent MDE at 80 percent power would demand roughly 30 times this sample. The required n grows with the square of the ratio of effect sizes.

Run length follows from both rules above. Kettle's behavioural cycle is a week, and two weeks covers two of them. Novelty is unlikely to drive a fix to a save interaction. There is nothing new to explore, only a control that works or does not, so you would expect the effect flat across both weeks rather than decaying. Check it anyway by splitting the estimate by week. If the lift decays, you measured novelty. Extending past two weeks buys precision only to the extent rho is below 1. If accounts are highly consistent week to week, a longer run buys calendar coverage, not power, at the cost of holding the arm longer.

A better primary metric is sitting right there, and proposing it is what reads as senior. Weekly cook rate runs the effect through two later steps the fix does not touch, each adding noise and carrying no signal. Make save-given-plan among web plan starters the primary. Two weeks of eligible web traffic supplies 1,747 plan starters per arm at the rho = 0 end. That detects an 8 percent relative lift, so the 25 percent target is powered several times over. At the rho = 1 end you have 873 independent starters per arm, which detects about 12 percent. That is still comfortably inside the 25 percent target.

That primary conditions on plan start, which is measured after randomization, so it needs the exception argued rather than assumed. The argument: the fix lives strictly downstream of plan start, in the save interaction, which a user only reaches after starting a plan. A change to a screen you have not yet seen cannot alter whether you get there. The check that proves it is cheap and you should name it: compare plan-start rates across arms and confirm they are balanced. If they are not, the change leaked upstream and the conditioning is unsafe. Fall back to saves per randomized web account, which measures the same thing with no conditioning at all.

Keep weekly cook rate as the secondary business read, and say plainly that alone it is thin here. Naming your own underpowered metric before the interviewer does beats a confident number that dies on the first follow-up.

SQL you would actually run

Interviewers often ask for the funnel query. Keep it flat and readable.

SELECT
  country,
  COUNT(*)                                          AS accounts,
  AVG(opened_app)                                   AS open_rate,
  SUM(started_plan) * 1.0 / NULLIF(SUM(opened_app), 0)   AS plan_given_open,
  SUM(saved_recipe) * 1.0 / NULLIF(SUM(started_plan), 0) AS save_given_plan,
  SUM(logged_cook) * 1.0 / NULLIF(SUM(saved_recipe), 0)  AS cook_given_save,
  AVG(logged_cook)                                  AS weekly_cook_rate
FROM kettle_account_week
WHERE week_start = DATE '2026-08-17'
GROUP BY country
ORDER BY accounts DESC;

The NULLIF guards are not decoration. A segment with zero accounts at a step divides by zero and takes down the report, and an interviewer who has been on call will notice you handled it.

---

The marketplace variant: when the two sides pull apart

On a marketplace the same prompt arrives as: one category is underperforming, is that demand or supply. Loomlane sells handmade home textiles, wool rugs are the weak category, and weekly sales are flat while the rest of the catalogue grows.

This is hard because the two sides create each other. Buyers show up where selection is good; sellers list where buyers are. Whichever side you measure, its weakness is partly an echo of the other's. So "conversion is low, therefore supply is bad" mistakes an equilibrium for a cause.

Split the funnel at the point where intent is established

Find the moment a buyer reveals intent, then treat everything before it as demand and everything after as supply. On a search-driven marketplace that moment is the search. Someone who typed "wool rug" and applied a filter wants a wool rug. What happens next is a referendum on the inventory.

The last row gives an answer independent of your own site. Run a small ad campaign on category terms. Healthy click-through means people want wool rugs and whatever is wrong is on your side of the door. Weak click-through against competitive creative means the category may simply be small, and no amount of seller recruitment fixes that.

The diagnostic that separates them

Sharpen the intent definition and watch conversion respond. This is the most useful single move in the whole variant.

Start with everyone who searched the category. Narrow to people who applied at least one filter. Narrow again to people who filtered and spent over 90 seconds in the category. Each narrowing removes browsers and keeps buyers.

Run the ladder on the weak category and on the whole catalogue at once, and compare the top-to-bottom ratios rather than the levels. The comparison is the diagnostic; a rise on its own is not. Conditioning on filter use and dwell selects on a correlate of purchase, so conversion climbs in every category on every marketplace, healthy or broken. The ratio is the readable statistic because it barely moves with inventory quality and moves a lot with how diluted the traffic is. The benchmark must come from the same platform, since the ratio also reflects how sharply your own filter and dwell screens separate browsers from buyers.

Say it with numbers. In wool rugs conversion goes 4.1 to 4.8 to 5.2 percent as intent tightens, a 1.3x rise. Catalogue-wide the same ladder goes 6.8 to 13.1 to 21.4 percent, a 3.1x rise.

Now write the inference out, because "flat, therefore supply" hides three steps.

Step one: the screens work. Filter use and dwell demonstrably separate browsers from buyers on this platform, because running that exact ladder catalogue-wide triples conversion. So a flat ladder in one category is not a broken screen.

Step two: given the screens work, a flat ladder means there were no browsers left to remove. If wool rug traffic were diluted with casual browsers, tightening intent would strip them out and conversion would climb the way the catalogue's does. It does not climb. So the people searching wool rugs already wanted a wool rug.

Step three: a 4.1 percent conversion among a population that was already high-intent cannot be blamed on the traffic. What is left is what those buyers found: assortment, price, shipping, quality. That is the supply side.

The steep case reads the other way and is worth having ready. A ratio much steeper than the benchmark means the base population was diluted. The low level is then low-intent traffic hitting inventory that serves real buyers fine. That is demand quality, and it is fixed in acquisition and merchandising. Flat is the stronger of the two readings, because selection should mechanically produce a rise and its absence is evidence by itself.

Supply is also the more expensive verdict, which is why you want the elimination written down before anyone commits budget. Recruiting sellers and filling assortment gaps runs on quarters: you have to find makers, onboard them, and wait for listings to appear and mature. Acquisition and merchandising changes run on weeks: a creative swap, a bid change, a ranking adjustment.

Filter usage then localizes it. Heavy price-filter use with the lowest conversion of any filter cohort points at inventory priced above what this audience pays. Heavy size or colour filtering with a high zero-result rate points at assortment gaps. Heavy shipping-speed filtering points at fulfilment, which neither more listings nor lower prices will fix.

Interview tip: Say "I would tighten the intent definition and see whether conversion responds relative to the same ladder run catalogue-wide" as an explicit diagnostic step. The comparison clause is what stops the interviewer answering with "steeper than what?"

Why a fix on one side breaks the other

Candidates who have only seen one-sided products come apart here. Every lever on a marketplace has a sign on both sides, and the signs often disagree.

Two consequences follow.

Every marketplace proposal needs a guardrail on the opposite side. Test buyer-side promotion of cheap listings and the guardrail is seller-side. Watch active seller count, share of gross merchandise value from sellers outside the top decile, and seller churn at thirty days. A buyer-side win with a seller-side regression is a loan, not a win.

The measurement window has to outlast the seller's reaction time. Sellers respond to last month, not last Tuesday. A two-week test can show a clean buyer-side gain and miss the delisting that arrives in week five. That is the short-term versus long-term tension covered later in this section, and marketplaces are where it bites hardest.

Liquidity as the arbiter

When the sides genuinely conflict you need a number both live inside. That number is liquidity, in a buyer version and a seller version.

• Buyer liquidity: share of category searches that end in a purchase within seven days.

• Seller liquidity: share of listings created that sell within thirty days.

Liquidity arbitrates because you cannot push one side indefinitely at the other's expense without the pair falling. Flooding the category with cheap listings raises buyer liquidity briefly and craters seller liquidity, since each seller's odds of selling drop as the pile grows. Recruiting only premium sellers does the reverse. A change that raises both is real category growth. A change that raises one and lowers the other is a transfer, and you should name which side you are taxing and why.

Read the two together and you get a two-by-two that names the diagnosis for you.

The two shortage cells are what most people jump to, and each is ruled out by a low reading where a shortage would have produced a high one. A genuine supply shortage cannot produce low seller liquidity, because scarce listings all sell. A genuine demand shortage cannot produce low buyer liquidity, because the few buyers who show up have their pick. Only the both-low cell eliminates both shortages at once, and what is left is matching.

For Loomlane: buyer liquidity in wool rugs is 4.1 percent against 6.8 percent catalogue-wide, and seller liquidity is 31 percent against 44 percent. Both low, which lands in the bottom-right cell and points at a matching problem rather than a shortage on either side. So I would check the zero-result rate on size and colour filters before spending anything on acquisition or seller recruitment.

---

What the whole answer sounds like in ninety seconds

Compressed, so you can rehearse it.

"I would define engagement as the share of 28-day-active accounts that log at least one cooked meal in a week, because cooking is what the product exists to cause. Today that is 11.2 percent. I would break it into four conditional steps, open, start a plan, save a recipe, log a cook, and then compute those four rates inside each segment. That grid shows three isolated weak cells: India loses only at plan start, web loses only at recipe save, paid social loses only at the final step. I would size each as extra weekly cooks after multiplying through the downstream rates, which gives roughly 470, 130, and 180 against a base of 6,700. I would ship the web fix first, because it is one engineer-week and the mechanism is nearly certain. Then the paid social targeting change, then scope the India work, which is the biggest prize and the largest build. For the web fix, randomize eligible web accounts. The primary metric is save-given-plan among web plan starters, because that is the step the fix touches, with weekly cook rate as the secondary business read. Guardrails are cook-given-save, support contacts and 7-day return rate. I would set the minimum detectable effect from the roughly 25 percent relative lift the sizing already implies, which two weeks of eligible web traffic detects with margin. So the test runs two full weeks."

Nothing in it is a feature brainstorm, and no metric appeared partway through. Every number came from the decomposition.

---

Common traps

Answering with a feature list. The most common failure. Fix: never name a feature before you have named the weak cell it addresses.

Confusing improve with diagnose. Fix: ask one clarifying question at the top, "has this declined, or is it just below where we want it," and take the right branch.

Ranking by rate gap instead of volume. A segment 20 points below average that contains 400 accounts is not the priority. Fix: always convert a rate gap into accounts, then into metric units.

Forgetting downstream conversion when sizing. A middle-step fix does not deliver its full gain to the end. Fix: multiply by the segment's own downstream pass rates, as the sizing table in Step 5 does.

Proposing changes to context. Country, tenure, device ownership, and seasonality are slices, not levers. Fix: for each proposal, name the specific thing an engineer would change.

Gaming the metric with a mix shift and calling it growth. Cutting your worst acquisition channel raises the rate and can lower the total. Fix: report absolute count alongside every rate.

Treating a marketplace as one funnel. Buyer conversion in a weak category is not a supply verdict on its own. Fix: split at the intent point, tighten the intent definition, and check whether conversion responds relative to the same ladder run catalogue-wide, since tightening intent raises conversion everywhere.

Shipping a one-sided marketplace fix. A buyer-side lift with no seller-side guardrail is borrowed. Fix: pair every marketplace proposal with an opposite-side guardrail and a window long enough for the other side to react.

Attributing a mechanism to a marginal cell. A weak cell in one dimension can be an artifact of the segment's mix on a dimension you did not slice. And a confounder reaching only one step looks isolated. Fix: recompute the weak rate inside the other dimensions against the same reference group before proposing.

Conditioning the analysis on post-treatment behaviour. Filtering both arms on something the change could have moved replaces your randomized comparison with a comparison of two self-selected groups. Fix: ask whether the treatment could have moved the variable, and if it could, argue the exception explicitly or drop the conditioning.

Sizing a test for an effect the population cannot deliver. A small round minimum detectable effect sounds conservative and often leaves you at single-digit power. Fix: size for the effect the proposal is buying, then print the achievable MDE for your eligible sample and say which one binds.

Changing the definition mid-answer. Once you drift, nothing you computed earlier still applies. Fix: write the definition down at the start and refer back to it by name.

---

Quick self-check