1.3 Diagnosing Metric Movements and Finding Bugs

Suppose the weekly engagement number falls twenty percent at 9am and a leadership review starts at 2pm. The investigation must distinguish a product regression, a measurement problem, and a change in the population being measured. Each explanation calls for a different response. This lesson shows how to use those five hours: verify the calculation and data pipeline, compare the right populations, then investigate the behavior that remains unexplained.

Why this matters in interviews

Section 1.1 asked how to improve a metric when nothing had necessarily broken. Here the task is to explain an observed movement. Start by checking whether the number represents a real behavioral change. Generating product hypotheses before that check can send the investigation toward a feature or user segment when the cause is in measurement.

Use four questions to organize the diagnosis. They are a practice checklist, not a claim about an employer’s scoring rubric.

Do you check the plumbing before the story. A large fraction of dramatic metric movements are not behaviour at all. They are a deploy, a schema migration, a bot filter that started or stopped working, a timezone, or a partial week of data. A candidate who spends ten minutes on user motivation and never asks whether the pipeline is healthy has revealed they have never been on call.

Do you decompose before you hypothesise. There are usually a hundred plausible stories and three arithmetic possibilities. Narrowing the arithmetic first turns a hundred stories into two.

Do you separate mix from rate. This is the highest-value skill in the lesson. "Mix" means which segments the users come from; "rate" means how those users behave. An aggregate rate can fall while every subgroup rises. If you cannot articulate why, you will ship a fix for a problem that does not exist.

Do you know a description from a cause. Finding that the drop is concentrated in one country is a description. Believing that the country caused it is the mistake.

Interview tip: Open with one sentence that names the branch you are on: "before I hypothesise about users, I want to establish that this is a real change in behaviour rather than a change in measurement or in population." That sentence alone separates you from most of the pool.

Everything below runs on a fictional product called Tessera, a photo and short-clip sharing app. The headline metric is reactions per active user per week: total reactions divided by weekly active accounts. Week 23 to week 24, it fell from 8.48 to 6.69, a drop of 21 percent. That drop has three separate pieces with three different owners, and none of them is the engagement regression the headline implied.

---

A dataset to make this concrete

Every number below comes from an account-week table. One row is one active account in one week. There are two weeks, 40,000 rows in week 23 and 52,000 in week 24.

The table hides three things and announces none of them. Finding all three, and routing each to the owner who can actually act on it, is the exercise.

---

Step 1: prove the movement is real before you explain it

Numerator, denominator, or both

Every ratio metric has two independently owned halves. Reactions per active account is total reactions / active accounts, and those two numbers are produced by different systems, moved by different teams, and broken by different failures. So a ratio can fall four ways: the top fell, the bottom rose, both moved unfavourably, or both moved the same direction by different amounts. Candidates habitually assume the first. Check.

That reframes the problem. Total reactions went up 2.5 percent while the account base went up 30 percent. The numerator rose, so the aggregate fall is not coming from the top of the ratio.

Be precise about what that does and does not establish. It does not mean nobody reacted less. Total reactions rising is perfectly compatible with large groups of individuals reacting less, and Step 3 finds exactly such a group. What it establishes is where to look: the denominator grew by 12,000 accounts, and whether that is bad news depends entirely on who those accounts are. That is a growth question, not an engagement regression.

A lot of people carry the wrong picture of the mechanism. Reactions are not a fixed pot being shared out among more people. Each account produces its own. The average falls when the accounts you add react below the current average, and it rises when they react above it. Growth on its own does not dilute anything. Growth of a below-average segment does, and we are about to find one: the incoming accounts react at 1.98 against a base of 8.48.

The same logic answers the year-over-year variant. If the metric is this year divided by last year, a decline can mean this year fell or last year rose. A promotion that inflated the same week twelve months ago shows up today as a fall, and no amount of staring at current-quarter product changes will find it.

Interview tip: Say the words "numerator and denominator" in the first minute of any metric-movement question, then state which one you found moved. This reads as senior because it routes the investigation. A denominator move is a population or eligibility question and belongs to growth. A numerator move is a behaviour or logging question and belongs to product. Naming which half moved hands the problem to the right team before anyone has told a story about it, and that is the whole reason the habit is worth eight seconds.

Is it outside the noise band

Mind which quantity you are testing. The standard error on each weekly mean is about 0.020 reactions per account. But what moved is the difference between two weeks, and a difference carries the noise of both weeks, so its standard error is sqrt(se23**2 + se24**2) = 0.028. The 1.79 movement is 63 standard errors, not the 89 you get by dividing through a single week's standard error. Quoting a gap in standard errors just means dividing it by its own standard error, which is the t-statistic you would report; past about 2 it sits outside the noise. Say that and move on.

Do the same arithmetic for smaller movements, because it usually kills the investigation early. 40,000 accounts with a per-account standard deviation of 4.0 gives a standard error of 0.020 on each week's level. That produces two different 95 percent bands, and confusing them is the classic error:

• Level band, for plotting each week's value: 1.96 x 0.020, or plus or minus 0.039.

• Move band, for plotting week-over-week changes: 1.96 x 0.020 x sqrt(2), or plus or minus 0.055.

A move of 0.03 is a chart artefact. Even 0.05 is not yet news. The sqrt(2) assumes the two weeks are separate draws, which is the usual case when the population is growing. If you track one fixed cohort against itself, you pair the observations and the band is tighter.

For a metric you diagnose repeatedly, precompute both bands once and label which is which. Putting the level band on a change chart is the usual mistake. It is anti-conservative in a way the two numbers make obvious: the 0.039 band is narrower than the 0.055 one. Drawing it on deltas therefore flags every move between 0.039 and 0.055 as real, when none of them is. You end up explaining noise. A large share of "why did this move" pages are answered by a control band nobody drew.

Is the comparison window honest

Six ways two windows fail to be comparable, and the check for each:

Only after those come back clean should you talk about users. In an interview you name them in fifteen seconds and say which you would run first given the shape of the move.

---

Step 2: split the change into mix and rate

Any average over segments is a weighted sum: the overall rate equals the sum of (segment share) times (segment rate). Two kinds of number feed it, so a change between two periods has exactly two sources: the rates moved, or the shares moved. Usually both. The game is attributing how much to each.

Write the baseline overall rate as R0, segment shares as w, segment rates as r. Subscript 0 is the baseline period, 1 the current one. Then

• Rate effect for segment i is w_i0 * (r_i1 - r_i0): what would have happened if only the rates had moved, holding the population fixed.

• Mix effect for segment i is (w_i1 - w_i0) * (r_i0 - R0): what happens because the segment grew or shrank, scored against how far above or below the overall average it sat.

• Interaction is (w_i1 - w_i0) * (r_i1 - r_i0): the part where a segment changed size and changed rate. It is normally small, and you report it so the three add exactly to the total.

Centering the mix term on R0 is what makes the per-segment numbers mean anything. Take the concrete case. Without centering, the mix term is just (w_i1 - w_i0) * r_i0. Every growing segment then scores positive and every shrinking one negative, no matter how good or bad the segment is. India grew and PH grew; both would look like gains. With the centering, a growing segment drags the average down only when it sits below the average, which is what you actually mean by "the mix got worse".

Reading the output

Run that decomposition over Tessera's five countries, week 23 as the baseline and week 24 as the current period, and it lands here. Shares are each country's slice of weekly active accounts, rates are its reactions per account, and the three effects are in reactions per account.

Read each row across and the three effects add to the total, which is the point of carrying the interaction column. Drop it and PH appears to be off by 0.009 and the US by 0.024, and a reader will assume you made an arithmetic mistake.

The entire drop is mix. Rate effects sum to plus 0.017, which means that if the population had held its week 23 composition, reactions per account would have been very slightly higher.

That last sentence has a name and a one-line computation, and it is the most important number in the lesson. The technique is direct standardization: take one period's rates and score them under the other period's population shares, so the only thing allowed to differ is behaviour. Here you take week 23's country shares, w0 = w23["country"].value_counts(normalize=True), and week 24's per-country rates, r1 = w24.groupby("country")["reactions"].mean(), and sum the product, (w0 * r1).sum(). That gives 8.5015 against a baseline of 8.4843, a change of plus 0.2 percent. It is the same quantity the table already computed a different way: the baseline 8.4843 plus the summed rate effects 0.0171 lands on the same 8.5015. Raw down 21 percent, standardized up 0.2 percent.

What happened is that the Philippines went from 2,399 accounts to 16,213, and those accounts react at 1.98 against a company average near 8.5. Someone bought traffic.

Two different counts of "new" are in play here, and it is worth separating them once. They do not match, and that looks like an error if you have not been told. The account base grew by 12,000 net, from 40,000 to 52,000. Separately, 22,167 week-24 rows carry tenure = "new", against 10,373 in week 23. Gross acquisition exceeds net growth because accounts also churn out. Of those 22,167 newly acquired accounts, 14,305 are in the Philippines.

Interview tip: Always report the mix-adjusted change next to the raw change. "Raw down 21 percent, mix-adjusted flat" is a complete finding in nine words, and it immediately redirects the room from the product team to the growth team.

The recommendation is not "fix engagement". It is: report the acquisition cohort separately from the established base, then ask growth what the campaign optimised for. A channel delivering accounts that react at a quarter of the company rate is either badly targeted or measured against the wrong objective. If the vendor was paid on installs, they did their job and the objective is wrong.

---

Simpson's paradox is mix shift with the sign flipped

Simpson's paradox is the extreme case of the same arithmetic: every segment moves one way and the total moves the other. It is not a curiosity. It is a weekly occurrence in any product with growing segments of unequal quality.

The Tessera numbers are one step short of it. Four of five countries improved and the total fell hard. The fifth is the US, and it fell for a reason that has nothing to do with the mix shift: a regression confined to US web, which Step 3 uncovers. To turn this dataset into a textbook Simpson case, you would have to remove that US web regression so all five country rates rise. Only then would you push the share shift. Growing PH further, or making the incoming PH accounts react even less, would deepen the total and enlarge PH's own contribution. But neither touches the US rate, so neither flips the fifth country.

concept flow

Why the aggregate can move against every part

1. 1
   Segments differ in level

   PH sits at 1.94, US at 10.48, so which segment a user belongs to matters more than anything that happened to them this week

2. 2
   Shares move faster than rates

   rates drifted by 2 percent, the PH share moved from 6 percent to 31 percent

3. 3
   The average is a weighted sum

   reweighting toward a low segment lowers the total even if that segment improved

4. 4
   The paradox condition

   share movement times level gap exceeds the sum of within-segment gains

5. 5
   The fix is not statistical but reportorial

   publish the standardized rate beside the raw one, permanently

The three defences, in the order you should offer them:

1. Standardise. Hold the weights fixed at a reference period and report the resulting rate. Demographers call this direct standardization; it is the (w0 * r1).sum() shown two sections ago.

2. Report the segments. A number that mixes a mature base with a promotional cohort is a number nobody can act on.

3. Redefine the population if the mix change is permanent. Suppose Tessera is genuinely now a company with a large Philippines base. Standardising to last year's mix forever then becomes its own lie. Change the definition deliberately, announce it, restate history.

The trap in the third option is that it doubles as perfect cover for a real regression, since somebody will always argue a decline is "just mix". Which is why you do the next step regardless.

---

Step 3: real change or logging change

You have shown the aggregate is mix. That does not mean nothing broke. Run the decomposition one level deeper and look for anything that fails to fit the pattern.

There it is. US web fell 29.5 percent while US mobile rose. There are 1,968 such accounts, costing about 6,100 reactions per week, a rounding error against 348,012 and invisible in the headline.

Do not assign it yet. You now have to decide whether this is broken measurement or changed behaviour. Provisionally it is the only finding that belongs to a product team, but "provisionally" is doing real work in that sentence.

Telling a broken metric from changed behaviour

Before filing a bug, decide which kind of failure this is. The signatures differ.

Now run it against US web out loud, because a table you never apply teaches nothing. Floor: it landed on 7.43, not zero and not a round number, and 99.7 percent of those accounts still reacted at least once (behaviour). Shape: a partial drop of 29.5 percent, not a step to nothing (behaviour). Breadth: confined to one client and one region, which points the other way, at instrumentation. Companion metrics: US android and ios moved plus 1.6 and plus 2.1 percent in the same country over the same week. So this did not move everything from the same log by an identical amount (behaviour).

That is four signals for behaviour and one for instrumentation, which is a lean, not a verdict. So spend two probes before you hand it to anyone. Pull a raw reaction event from an affected web account and confirm the rows exist at all. Then check whether the error rate on the reaction endpoint moved for web clients in that window. If the events are there and the endpoint is clean, it is a behaviour regression and the product team owns it. If either probe comes back dirty, it is a client or pipeline bug and engineering owns it. Two queries, and you stop guessing.

An exact zero for a well-defined subset is a strong reason to inspect instrumentation and the user-facing path. If 14,000 accounts previously performed an action and none now records it, check whether the action is blocked, events are missing, or a filter excludes them. The zero alone does not prove a logging defect: a genuine outage or a rule preventing the action could also produce it. Reproduce the action and compare independent records before assigning the cause.

Read the table rather than the sketch above, which shows more grain than the data has. The data is two weeks at weekly grain with one row per account, and there are six platform-by-build cells, not four. Uploads per account on Android build 8.4.0 are exactly 0.0000, while the same build on iOS and web is normal and the previous Android build is normal. That is the third thing hidden in the data, and it is a bug with a two-dimensional fingerprint: one platform, one version.

One detail is worth more than it looks. Build 8.4.0 already existed in week 23, on 1,252 Android accounts, uploading a perfectly normal 0.7676. The uploads did not stop when the build shipped; they stopped at the week boundary. That rules out "the release broke it on day one". It sends you looking for what else changed that week: a server-side flag, a config push, a backend deploy the client build only became sensitive to later.

In the previous week every one of those cells sat between 0.7676 and 0.8243.

Notice too that the unaffected cells drifted from about 0.78 into the 0.69 to 0.81 range. That is the mix shift again, quietly moving a second metric. Two phenomena in one table, and you must name both or you will over-attribute the bug.

Interview tip: When a metric hits exactly zero for a subgroup, say "this is a code path, not a behaviour change" immediately and spend your remaining time on where in the stack it broke. Interviewers use the zero specifically to see whether you notice.

The four layers where a number can break

A dashboard number has travelled through four independent systems and any one can be the culprit. Enumerate them out loud; it is a two-sentence answer that sounds like experience.

checklist

Bisecting a broken metric, outermost first

• The user-facing path: can a real person on that build complete the action, tested by hand on the affected client

• The emission path: the action succeeds but the client never fires the event, or fires it with a field the pipeline drops, so no row lands

• The transform path: rows land and the aggregation loses them, via a join that went inner, a filter on a renamed enum, an over-collapsing dedupe key, or a partition never written

• The presentation path: the table is right and the chart is wrong, via a stale materialized view, a cached tile, a bad date filter, or a deprecated column

Three of those terms are data-engineering vocabulary. A dedupe key is the set of columns a pipeline treats as identifying one event, so a key that is too coarse merges distinct events into one and quietly deletes rows. A materialized view is a query result stored as a table, which keeps serving yesterday's answer until something refreshes it. A partition is one slice of a table, usually a day, which a job can fail to write without failing loudly.

Work outside in. The outer layers are cheapest to test and the most damaging if true: a broken upload button costs money right now, a broken dashboard costs only credibility. Two decisive probes. Pull a single raw event for an affected account and see whether it exists. Then check whether an independent system agrees. If uploads are zero but object storage still shows new blobs arriving from that build, the product is fine and the logging is broken. You have localised the fault in one query.

If asked how to catch these sooner: user-facing breakage produces a support signal before a dashboard signal, because affected users complain in minutes while a weekly metric takes days. Classify inbound support text by product area, alert when an area exceeds its own trailing baseline, route it to the owning team. Cheap, and it routinely beats the analytics stack by a day.

---

Step 4: when the change is real, describe who changed

This technique answers "who is different this week", and it serves both branches of the investigation. On the mix branch it names the incoming cohort. On the real-change branch it names the population the regression is hitting. The worked example below is the first kind: we already know PH is the story, and the point is to watch the method recover it without being told.

Suppose the mix shift were not obvious from a single dimension. With twenty candidate dimensions and interactions between them, hand-slicing is slow. The standard trick turns "who is different this week" into a supervised learning problem. Label current-period rows 1 and baseline rows 0, then fit a model to predict which period a row came from. If a model can tell the weeks apart, whatever it splits on is what changed about the population. Use a shallow decision tree: a model that repeatedly splits the rows on one variable at a time. Prefer it here because you can read its rules directly instead of interpreting coefficients.

Fit one to depth 2 on country, platform and tenure, with a floor of 800 rows per leaf, and it carves the 92,000 rows, both weeks pooled, into four groups. The base rate to compare each leaf against is 56.5 percent, week 24's share of all rows.

The first split is country_PH. The node on the right of that split holds 18,612 rows that are 87 percent week 24 and 88 percent new accounts, against a base rate of 56 percent. That is the campaign, recovered automatically without anyone naming the Philippines.

Read the node, not a leaf, because a depth of 2 splits that node again, on platform_android, into leaves of 3,079 and 15,533 rows. Both leaves are about 87 percent week 24, which tells you the second split bought nothing; the whole signal was the first one. Stop at depth 1 and the 18,612 rows become a single leaf.

A tree suits this job for three reasons. It handles correlated inputs without the instability that wrecks a regression's coefficients. It produces rules you can paste into a slide. And in general it gives interactions free, because a leaf is defined by every split above it. "PH and new" would be one leaf, rather than two separate effects you have to multiply by hand.

That third property is a property of trees, not a demonstration in this fit. This tree never splits on tenure at all. Its three splits are country_PH, country_DE and platform_android. Tenure loses inside that node rather than overall. A split is chosen by how much impurity it removes among the rows that actually reach it, which is a different question from how well the feature scores across the whole frame. You will see tenure's standalone 0.583 in a moment, above platform's 0.543, and it still does not win the slot: the standalone score is not what the tree is optimising. If you want to see the interaction property actually fire, raise the depth or drop platform from the feature list.

The leakage that ruins this every time

Add app_version to the feature list and the tree stops being useful. Its first split becomes version 8.4.0, which sorts the weeks far better than any real cohort variable and tells you nothing, because a release rolled out during the comparison window. Six percent of baseline rows and 75 percent of current rows carry that build. The model has found a clock, not a cohort.

Any feature that is a function of time will dominate this model and teach you nothing. Your primary defence is a drop list written before fitting. The discriminator is not "does it correlate with the period" (every useful feature here does), but where that correlation comes from. Drop variables whose period-correlation is produced by the measurement apparatus: a rollout, a schema change, a collector switched on last month, anything derived from the event timestamp. Keep variables whose period-correlation is the population change you are trying to describe.

Tenure falls on the keep side, which is why it stays in the feature list despite being a days-since-signup bucket. Its share of new accounts moves from 25.9 percent in week 23 to 42.6 percent in week 24, the 10,373 and 22,167 new accounts counted earlier over row totals of 40,000 and 52,000, and that shift is not an artefact of how Tessera measures anything. It is the campaign, which is the finding. App version's shift, by contrast, is a release train. Both are correlated with the period; only one is about users.

The backstop is to score each feature alone on how well it predicts which period a row came from.

Anything clearing about 0.75 alone, or beating every real cohort variable by a wide margin, is a clock. Score AUC or balanced accuracy rather than raw accuracy. AUC is the probability that a randomly drawn week 24 row scores above a randomly drawn week 23 row, so 0.5 is a coin flip; it is the area under the ROC curve, which traces true positives against false positives as you sweep the cutoff. Raw accuracy is hard to read against a shifting majority baseline: predicting "week 24" for every row already scores 56.5 percent here. That floor moves whenever the two periods differ in size. The rule "predict w24 if and only if the row is on 8.4.0" scores 83.4 percent raw. Its balanced accuracy, the average of how often it gets each week right, is 0.846, and its AUC is also 0.846. Those are the numbers to compare against the 0.75 line. They say the separation is real rather than an artefact: 94 percent of week 23 rows sit on 8.3.1, against 75 percent of week 24 rows on 8.4.0.

Treat the number as a backstop and the drop list as the defence. A naive 90 percent accuracy screen would wave app_version straight through. A purely relative rule fails the other way, since a genuine cohort shift big enough to dominate everything else trips it and you bin the finding you wanted.

Two guardrails. Significance is meaningless here because n is enormous and every feature will clear any threshold, so rank by lift and leaf size, never by a p-value. And this model describes, it does not explain. That the new accounts are Filipino and new is a description; the cause is a purchase order somebody signed. The model narrows where to ask, a human still answers.

---

Self-selection: when moving A does not move B

Now the failure mode that costs the most engineering time. Tessera observes that accounts with a profile photo react far more than accounts without one.

A 31 percent gap. The obvious product move: auto-generate a photo for everyone, or nag until they upload one, and harvest the engagement. It will not work, and you should be able to say why in twenty seconds.

Setting a profile photo is a choice, and the choice is made by people who already care. It is a symptom of intent, not a producer of it. Controlling for tenure shrinks the gap from 31 percent to 21 percent among new accounts and 14 percent among returning ones.

The tempting next sentence is that the remainder is the part of caring that tenure does not capture. It is wrong here, and worth catching, because the remainder is a variable that does not appear in that table at all: country. The two photo groups live in different countries. Accounts with a photo are 45 percent US and 2 percent PH. Accounts without are 25 percent US and 12 percent PH, and PH reacts at a fifth of the US rate. So standardise both groups onto the joint distribution of country and tenure, the same move used for platform a section from now. Score each group's country-by-tenure cell means under the whole base's cell shares and both land on 8.48 reactions per account.

Neither control alone is enough, which is the real lesson. Tenure alone leaves 14 to 21 percent, country alone leaves 10 percent, and jointly the 31 percent gap goes to zero. All ten country-by-tenure cells have photo and no-photo means you cannot tell apart, the widest gap 1.7 standard errors. US returning is 11.41 with a photo against 11.45 without. IN returning is 6.88 against 6.88, and PH new is 1.82 against 1.82. Profile photo carries no information that country and tenure do not already carry, which is what a pure marker of intent looks like once you have measured the intent.

One honest limit. The PH returning cell holds 54 accounts with a photo against 219 without, so its agreement carries almost no weight. The conclusion rests on the nine well-populated cells.

The product conclusion is unchanged, and now demonstrated rather than asserted. Assign the photo by fiat and you have moved the marker without moving anything the marker was marking. Most disengaged accounts will not notice. Some will be annoyed that you touched their profile, so the effect can be negative.

Do not carry the clean zero into a real table. On production data the residual rarely closes completely. The honest sentence has two parts, and their order matters. Some of the remainder is composition I have not standardised on yet. Some may be intent I cannot measure. Reaching for the second before exhausting the first is how a photo-nag project gets funded.

The general form recurs constantly: when a variable is produced by user choice, forcing it moves the label off the behaviour instead of moving the behaviour. Users who enable notifications retain better. Users who join a group retain better. Users who connect a second device retain better. Each is mostly self-selection, and each has had an engineering quarter thrown at it somewhere.

So what is the finding good for? Quite a lot, if you invert the problem.

concept flow

Turning a self-selected marker into something you can ship

1. 1
   Stop treating the marker as a lever

   profile photo is an outcome, so nothing is gained by forcing it

2. 2
   Promote it to a label only if it survives standardization

   a marker that still separates users after you control for composition is a cheap, dense, early stand-in for a slow outcome, but this one does not survive, so a model trained to predict it would recover country and tenure and nothing else

3. 3
   Restrict the inputs to things you control

   surfaces shown, latency, notification cadence, onboarding steps, ranking weights, not country or device

4. 4
   Rank what is left

   order the controllable drivers by effect size and by how sure you are the arrow points from input to outcome

5. 5
   Test the top one or two

   a model on observational data ranks candidates, it never certifies a cause

6. 6
   Keep the marker as a guardrail, using a marker that is not the outcome you are measuring

   if the marker rises and engagement does not, your intervention moved the proxy rather than the thing

The tell for a self-selected variable is a single question. Could a user have chosen not to have this value, and does that choice say something about how much they care? If yes, it is a symptom. Contrast that with something like assigned onboarding variant, which the user did not choose, and which is therefore a legitimate lever.

Interview tip: When you spot a self-selected driver, do not just say "correlation is not causation". Say the concrete failure: "we would set the flag without moving the underlying intent, so the association would break rather than transfer." Naming the mechanism is what scores.

---

Actionable insight or spurious relationship

The neighbouring failure mode: a variable the user did not choose that is still not a cause, because it stands in for something else. Take platform in the week 23 base.

Now split the same week by country and platform together. Every cell is reactions per account.

In aggregate, iOS accounts react 23 percent more than Android accounts. Within every single country the three platforms are indistinguishable. Platform is a proxy for country: 54 percent of iOS accounts are in the US, against 23 percent of Android accounts. And 29 percent of Android accounts are in India, where the base rate is 6.3.

The cleanest way to prove it, and the one to describe in an interview, is direct standardization again. Recompute one group's average under the other group's covariate distribution, so composition is held identical and only behaviour can differ.

Give the Android accounts the iOS country mix and they react at 9.55, against iOS at 9.49. The gap does not shrink, it inverts slightly and lands inside noise. Platform carries no information about reactions that country does not already carry.

Three tests, in increasing cost:

What if the variable survives all three? Then the difference really is attached to it, and your next move is mechanism. For a surviving platform gap, pull engineering telemetry rather than the user table. Look at cold start time, crash-free session rate, time to first render, error rate on the reaction endpoint, and the spread of OS versions still in the field. A real platform effect is almost always a quality effect, and quality lives in latency and errors, not demographics.

Interview tip: Ask "is X a cause, a proxy, or a symptom" as an explicit three-way question. What separates a proxy from a symptom is who set the value. A proxy is a variable the user did not choose (platform, device tier), so the fix is to find the confounder standing behind it. A symptom is one the user did choose, so after you exhaust composition it can still carry residual intent, and the fix is to treat it as a label rather than a lever. Only causes are levers. In this dataset the photo's residual is exactly zero, so it behaves like a proxy for country and tenure even though the real-world story is self-selection. On production data that residual is the number that tells you which of the two you are holding.

---

Behavioral versus demographic variables

Every investigation above eventually needs a set of candidate drivers, and you will be asked which family to reach for. Take Thistle, an online homewares retailer buying in-feed ad inventory on Tessera, wanting to predict whether a given user clicks a given ad.

The demographic family says: this account is 34, in Lisbon, has been registered three years, uses Android. The behavioral family says: this account viewed four rug listings yesterday, saved two, and searched "runner rug hallway" twice this morning.

Behavioral wins, decisively, for two reasons.

Timing. Demographics give the base rate for a type of person over a year. Behaviour says a specific person is in-market right now. Click-priced advertising is a game about the next thirty minutes, and a variable that resolves the base rate but not the moment is nearly worthless in it. That is why the industry's economics changed when targeting moved from one to the other.

Purchase intent is not personal fit. People buy for other people: gifts, a partner's request, a shared household account, an order placed for a parent. A demographic model implicitly assumes every purchase is for the buyer, and is wrong about a large slice of transactions. Browsing history does not care who the item is for. It records that this session is about rugs.

Where behavioral variables are weak is precisely where the earlier sections of this lesson bite.

Row one carries the prediction half: behavioral is "very high" against demographic's "modest". The Endogenous and Self-selection rows carry the catch, and they are the two sections you just read. Endogenous means the feature is partly the outcome itself: browsing a rug listing is already the first move of clicking a rug ad, so the model is partly predicting an event from its own opening step. The cold start in row two is the modelling one: a brand-new account has produced no behaviour yet, so a behavioral model has nothing to score until it acts. Together they give the sentence most candidates miss: behavioral variables dominate prediction, and being the strongest predictor is not the same as being a lever. They are easier to influence than demographics: the Assignable row says "sometimes, indirectly" against demographics' "never". But the influence is indirect, and weaker than the product-state variables you set outright. "This user browsed rugs" is a superb feature and a terrible instruction, since you cannot make a user browse rugs. What you can change is what you show, when you show it, and how fast it loads.

tradeoff matrix

Choosing a variable family for the job at hand

Choosing a variable family for the job at hand

Family	Strength	Weakness	Use when
Demographic and account attributes	Present at signup, stable, cheap, legible in a slide	Weak on timing, coarse, easy to over-interpret as causal	Cold start, eligibility rules, reporting cuts, fairness audits
Behavioral and session signals	By far the strongest short-horizon predictors	Endogenous, decays fast, largely self-selected, consent-heavy	Ranking, real-time targeting, propensity scoring, early-warning proxies
Product-state and assignable variables	Causal when you assign them at random, which is why they are what an experiment varies	Usually weaker predictors than behaviour	Anything you intend to ship, every experiment arm, every recommendation
Technical telemetry	Directly actionable, and a plausible mechanism to test rather than a certified cause	Narrow, only explains quality regressions, and still observational so it needs the same standardization as any other segment gap	A real segment gap survived standardization

Two labels in that matrix are worth a gloss. Propensity scoring means estimating each user's probability of taking some action and then ranking or targeting on that number. A fairness audit checks whether a model's errors fall unevenly across groups, which is why it reaches for the stable, legible attributes in the first row rather than behaviour that decays in hours.

Two rows in that matrix deserve a warning, because they are where people over-claim. Setting a variable by a product rule is not the same as assigning it at random. Give the new onboarding flow to everyone in one market and the flow is confounded with the market. Only randomisation buys you the causal reading. And telemetry is observational like anything else. Latency and crash rate track device tier and market. Android accounts here are 29 percent India, where the base rate is 6.3, the identical confound the platform section just dissected. A latency-engagement gap needs the same standardization as a platform-engagement gap before you believe it.

The rule: build with behavioral variables when the task is prediction, switch to assignable variables the moment the task becomes a recommendation. A model that ranks browsing history at the top of its importance list has told you it works, not what to build.

One second-order point, useful on product-design prompts. If behavioral data is what earns the revenue, then features that generate honest interest signal (saves, collections, follows) are worth more than their direct engagement lift. They feed everything downstream.

---

The ninety-second answer

Compressed, so you can rehearse the whole thing:

"Reactions per active account fell from 8.48 to 6.69, down 21 percent. First, numerator versus denominator. Total reactions actually rose 2.5 percent while active accounts rose 30 percent. So the fall is not coming from the top of the ratio. The base grew. Second, mix versus rate: decomposing by country, rate effects sum to plus 0.02 and mix effects to minus 1.81. The Philippines went from 6 percent of accounts to 31 percent, at a rate of 1.98 against a company average of 8.5. And 14,000 of the 22,000 newly acquired accounts are there, so this is an acquisition campaign, not an engagement regression. Standardised to last week's mix, the metric is flat. Third, I checked whether anything else hid under the mix. US web reactions fell 29.5 percent while US mobile rose 2 percent. That is small in absolute terms, about 6,100 reactions. But it lands untidily rather than at zero, and it spares mobile in the same country. I would treat it as a real regression after pulling one raw event and the endpoint error rate. And Android uploads on build 8.4.0 are exactly zero, which given the same build is fine on iOS is a client bug, not behaviour. So: three findings, one report split for growth, one web regression for the product team, one release bug for engineering. The headline number needs no fix, it needs a standardised twin."

No guesses about user psychology, and every claim is a number someone can check.

---

Common traps

Explaining a movement before confirming it exists. Fix: state the standard error on the difference, not on either level, and the comparison window first. A large share of pages are noise or a partial period.

Assuming the numerator moved. Fix: print both series. A ratio falling while the numerator rises is common and changes who owns the problem. Use the centered mix term (w1 - w0) * (r0 - R0) when you attribute, or every growing segment scores positive and the per-segment numbers mean nothing.

Ranking segments by rate change instead of contribution. A segment down 40 percent holding 200 accounts is not your problem. Fix: multiply by share, always report contribution in metric units.

Stopping at "it is just mix". Fix: after standardising, decompose one level deeper inside the largest segments. The US web regression is invisible at the top level and is the only finding here that belongs to a product team.

Treating an exact zero as a behaviour change. Fix: exact zeros for a well-defined subgroup are code, every time. Go straight to the four layers.

Letting a time-correlated feature into the who-changed model. Fix: drop anything whose period-correlation comes from the measurement apparatus. That means app version, event-timestamp derivatives, and fields the pipeline started collecting recently. Keep the ones whose period-correlation is the population shift itself, such as tenure.

Reading the who-changed tree as causal. Fix: say "this describes the incoming population" out loud, then go find the human who signed the purchase order.

Trying to move a self-selected marker. Fix: ask whether the user chose the value and whether the choice reveals intent. If yes, use it as a label, not a lever.

Accepting a segment difference without standardizing. Fix: recompute one group's average under the other's covariate mix before you write a single line of the recommendation.

Recommending a behavioral variable as an action. Fix: browsing history predicts, it does not instruct. Convert every finding into something the product actually sets.

---

Quick self-check