1.5 Feature Ideas, Similarity, and Predicting Behavior

Feature proposals, user similarity, product prioritization, and churn prediction all require a connection between observed behavior and a decision. This lesson works through those four tasks using the activity already recorded in a product's logs. You will define what the output represents, identify who can act on it, and explain how an analysis would support a test. In some cases, a simpler rule or a clearer product question is more useful than building a model.

Why this matters in interviews

The failure mode here is the opposite of the one in the metric-improvement lesson. There, candidates jump to features. Here, the question asks for a feature, and candidates take that as licence to be a product visionary for eight minutes: what users want, where the industry is going, what a competitor shipped last quarter. That talk is unfalsifiable, and it ignores the one asset the role gives you, a record of what millions of people actually did.

An interviewer is checking three things. Can you locate demand in behaviour that has already happened. Can you turn a fuzzy word like close, similar, or likely to churn into something with a numerator, a population, and a window. Can you say what the output is for, meaning what changes on someone's screen or in someone's queue the day it ships.

A churn model's AUC describes its overall ranking performance, but the intended workflow determines which part of that ranking matters. Suppose a retention team can contact 400 customers a day. Precision among the top 400 describes the quality of the queue that team can actually work. You still need to consider the cost and benefit of contact, which the later example develops, but naming the queue capacity gives the evaluation a concrete operating point.

The second answer wins for a mechanical reason, not a rhetorical one. A model emits a continuous score, and a score is not a decision. Somebody has to convert the ordering into a cut: work these rows, ignore the rest. Two things produce that cut in practice. One is a capacity limit, the number of rows a human or a system can process. The other is a price, the point where the expected value of acting exceeds the cost of acting. If you have neither, there is no operating point, and with no operating point there is nothing to optimise, which is why "AUC 0.66" hangs in the air. Naming the consumer is how you find the capacity. Naming the cost and the value is how you find the price. Later in this lesson you work the price route all the way through on one model, and find the capacity route never binds ahead of it.

Interview tip: Before you describe any model, say the sentence "the output of this is a ranked list that goes to X, who can act on Y of them per day." If you cannot finish that sentence, you have not been asked for a model.

If the honest answer is that nobody consumes the list on a fixed schedule, you need the other route: the price.

The three fictional products in this lesson are Wren, a group messaging app; Ember, a photo community; and Trailhead, a professional network.

---

A panel to make the modelling concrete

The prediction half runs on a monthly snapshot panel from Trailhead. One row is one member in one month. It holds only information observable at the end of that month, plus a label for whether that member started a new role in the following month. There are 216,000 member-months, 9,000 members across 24 months, and the label fires on 2.7 percent of rows.

Two rows of that table oversell what the panel holds, and you should know it before the numbers arrive. The months_in_role row promises a tenure hump, meaning a move rate that rises with time in role and then falls back. There is no hump here. Group the rows into tenure bands, from under six months to over six years, and the move rate reads 0.0268, 0.0262, 0.0267, 0.0258, 0.0269, 0.0283, 0.0285, 0.0311, 0.0300. That is flat, drifting slightly upward, never turning back down. Fit a model on that one column alone and it scores 0.511, which is a hair above chance. Read that cell as "weak, monotone, close to useless" rather than as a hump. The level and sector row is worse: both columns are drawn independently of the label. Move rates by level are 0.0284, 0.0268, 0.0274 and 0.0271 across ic1, ic2, ic3 and mgr, and by sector 0.0274, 0.0273 and 0.0274. That is noise with four and three labels on it.

This is a synthetic panel, and it departs from a real one in ways you should name yourself. A single hidden per-member trait drives both the four behaviour counts and the move probability, which is why behaviour will look so strong later. months_in_role climbs by one every month and never resets after a move. A member who took a new role in month 7 still carries seven more months in role at month 14 than at the move, because the counter never restarts. And nothing damps the hazard after a move, so members move in consecutive months: 4,009 of the 9,000 move at least once and 402 move three or more times. A real panel would reset tenure on the move and suppress the hazard afterwards. Treat this one as a rig for practising the mechanics.

The seasonal term is deliberate: hiring runs hot in months 1, 2, 12, 13, 14, and 24. The market column sits near 62 in every cold month and near 100 in every hot one, so those six months carry a genuine demand shock and the other eighteen do not. Split a panel like this at random and you train on the January you are about to be tested on.

---

Where feature ideas come from when you are not allowed to have opinions

The habit worth building: stop asking what users want, start asking what they already do the hard way.

The friction ledger

Every product contains jobs people accomplish today, but accomplish badly: a workaround, a copy-paste, a message to another human, a trip to a different app, six taps that could be one. Each is a demand signal somebody already paid for. So the ideation loop is not creative. It is forensic.

1. Enumerate the outcomes people reach today through effort.

2. Measure how much effort each one costs and how many people pay it.

3. Rank by volume times effort, which gives you a shortlist.

4. Divide that shortlist by how hard each shortcut is to build, which gives you the order.

5. Build the top item and test it.

Step three deserves precision, because the version of this rule you will hear in interviews, "people times effort", is not quite what anyone computes. Volume and people are different quantities. Below, the volume term is a share of closing messages, so an intent that a small number of members trigger constantly outranks an intent that many members trigger once. If the decision turns on how many distinct members are affected rather than on how much traffic the intent generates, you need a member count in the ledger. Either way, say out loud which one you are using. Step four is a separate step for the same reason: build cost is not in the log and never will be. Somebody has to score it.

Every step is checkable, so if the interviewer disagrees with your top idea they are disagreeing with a number, not with your taste. Keep the mirror image in your pocket too: to make people do something less, add steps. Teams do this deliberately, and naming it shows you read friction as a lever in both directions rather than as a bug.

Four grades of evidence

Not all demand evidence is worth the same. Interviewers rarely say this explicitly, but they grade your proposal by how directly the evidence proves that people already want the outcome.

The top row is the strongest thing you can say. The bottom row is what most candidates reach for. If you catch yourself citing a competitor, follow it immediately with the internal signal that would confirm the same demand on your own product, and go measure that instead.

Running the ledger on Wren

Wren logs every message and every tap, which is enough to turn the ledger into a pipeline.

Classify intent, not text. "Can you call me" and "free for a quick chat" are one intent, so embed each message, cluster the embeddings, and hand-label the clusters once. Weight opening and closing messages more heavily: openers carry why a conversation exists, closers carry what it produced, and the middle is mostly negotiation. Then measure the taps and app switches each intent costs today.

The table below is that pipeline's output, ranked. The friction score multiplies the share of closing messages an intent accounts for by the taps it costs beyond the first, and zeroes out anything already reachable in one tap. Taps minus one, rather than taps, because one tap is the floor: an intent that already costs one tap has no friction left to remove.

The bottom two rows are easy to skip past, and one of them is the interesting one. Seventh is "re-request a file" at 0.084. Eighth and last is "confirm receipt", at exactly 0.000.

Two things in that full ranking earn credit.

First, "confirm receipt" is the most common closing intent at 14.7 percent and scores zero. That is the ledger validating itself: read receipts already exist, so the intent is already one tap, and the already-one-tap column drives the product to zero. This is the sanity check that makes the whole score trustworthy. A scoring function that ranked a shipped feature first would be measuring popularity, not friction, and you would have learned nothing from it. Run this check before you show anybody the top of the list.

Second, the top raw score is not what you build first. Eleven taps means picking a restaurant leaves the app and comes back, which is a venue catalogue, a partnership, and a payments conversation. Live location is 98 of every thousand closing messages, costs seven taps, and is entirely self-contained inside Wren.

The ledger does not do this last step. It produces the friction column and stops. You apply the build-cost divisor by hand, and to make the ranking reproducible rather than asserted you have to write the scale down. Score each candidate 1 to 5. A 1 is a client-side change your own team ships in a sprint. A 5 means leaving the app for a third-party catalogue plus a commercial agreement. Pick a restaurant is a 5 and live location is a 2, and 0.610 divided by 5 loses to 0.588 divided by 2. Friction gives you the list. Build cost gives you the order, and it is the step that flips this particular answer.

From cluster to candidate

The last step translates an intent into something an engineer can build and a metric can see.

The last row carries two metrics on purpose. Broadcast makes sending cheaper, which mechanically raises messages sent while lowering how much recipients care. Pairing a volume metric with a quality metric is the cheapest way to show you thought past the obvious win.

Interview tip: When you propose a feature, name the exact log-derived number that proves people already want the outcome, and the exact number that would fall if the feature succeeded for the wrong reason.

---

Similarity: define the target before you pick a distance

"How would you find the two users who are closest" is a ranking question in disguise: given one item and a candidate set, order the candidates. Nothing about it is specific to social networks. Nearest podcast, nearest listing, nearest support ticket, nearest fraud ring, identical shape and a different candidate set. So the interesting work happens before any distance function appears.

Write down what closeness is supposed to mean

There is no true definition of best friend, most similar song, or comparable listing. There is only the definition that makes the downstream product work. State one in a sentence and accept pushback: pushback on a stated definition is a conversation, pushback on an unstated one is a failure.

For Wren: the contact a member would choose to spend an unplanned free evening with. That is deliberately about offline preference rather than message volume, because volume is dominated by logistics. The person you message most is often a group admin, a landlord, or the coworker who owns the schedule.

Different definitions imply different features. If closeness meant who influences whose purchases, you would weight shared link clicks. If it meant who would notice a compromised account, you would weight response latency to odd requests. Say which one you picked and why the product needs it.

User-user versus item-item

These get conflated constantly, and their economics genuinely differ. User-user similarity compares two people by what they do or who they know. Item-item similarity compares two pieces of content by who consumed both.

tradeoff matrix

Choosing the similarity you actually need

Choosing the similarity you actually need

Approach	Strength	Weakness	Use when
User-user	Captures taste directly, handles novel items	Users change fast, recompute often, sparse per pair	Social graph, close-contact ranking, lookalike audiences
Item-item	Stable over weeks, precomputable, cheap at serve time	Blind to why a user is here today	Recommendations, related content, catalogue navigation
Content-based	Works for brand new items with zero traffic	Only sees declared attributes, misses behaviour	Cold start, first 30 days of an item's life
Behaviour co-occurrence	Learns real substitution and complementarity	Popularity dominates unless you normalise	Mature catalogue with dense interaction logs

The practical rule: item-item is what you serve, user-user is what you analyse. Item neighbours move slowly enough for a nightly job, and there are usually far fewer items than users. A member's own neighbourhood shifts weekly, which is fine for a study and painful as a production dependency.

Structural overlap and why raw counts mislead

When your data is a graph, the first signal for the strength of a pair is how much their neighbourhoods overlap. It is the same intuition as co-occurrence in text: two things that keep appearing beside the same third things are related. So the naive score is the raw count of shared contacts.

Raw counts have two well-known problems, and the standard repairs both attack the same one. The first problem is scale: a member with 500 contacts shares more contacts with everybody, so raw counts make popular members look close to the whole network. Jaccard similarity fixes that by dividing the overlap by the size of the two neighbourhoods combined. The score then asks what fraction of everything these two people know is known to both, rather than how much is known to both in absolute terms. The second problem is that not all shared contacts are equally informative. Sharing an obscure contact who knows four people says far more about a pair than sharing a hub who knows four thousand. Adamic-Adar fixes that by weighting each shared contact by one over the logarithm of that contact's degree, so hubs contribute little and rare contacts contribute a lot. The logarithm rather than the degree itself is what keeps the discount gentle. A contact with 1,000 neighbours is discounted about twice as hard as one with 30, not thirty times as hard. A contact with exactly one connection has to be skipped rather than weighted, because log(1) is zero and the weight would divide by it.

Here is a small slice of a Wren graph in which Ana has two candidate close contacts, Bo and Zed.

Ana has ten contacts, Bo six and Zed five. Scoring the pairs on the three overlap measures, plus the number of distinct circles the shared contacts span:

Raw overlap, Jaccard, and Adamic-Adar all pick Bo, and Bo is a coworker.

Be precise about what that does and does not prove, because this graph is too small to be a fair test of either repair. There is no hub in it. Ana has degree 10, Bo 6, Zed 5, and every shared contact has degree 2 or 3, so Adamic-Adar has almost nothing to discount. Both unions happen to contain exactly 12 nodes, which makes Jaccard here nothing but the raw count divided by a constant, so it could not have disagreed. And Bo's shared contacts have degrees 3, 2, 2, 2 while Zed's have 3, 2, 2, so Bo's Adamic-Adar score is Zed's plus one over log 2, by construction. Neither measure was ever free to rank Zed first.

The stronger point does not need a hub. Jaccard and Adamic-Adar both normalise the count, and normalising a count still leaves you counting. None of the three looks at how the shared contacts relate to each other. Ana's four shared contacts with Bo could be four strangers or one tight team, and all three scores are identical either way. The signal that separates a coworker from a close friend lives in that structure, and no degree correction reaches it.

Circle breadth is the signal that separates them

The last column gets it right, and it is a different kind of measure: it reads the structure among the shared contacts rather than their number. Ana and Bo share four contacts from one circle. Ana and Zed share three spanning work, school, and family, which means something specific: Zed has been introduced across the boundaries of Ana's life and Bo has not. Sharing an org chart is not intimacy.

Start with each member’s ego network: their contacts and the edges among those contacts. Removing the member exposes the connections among contacts without the member’s universal links dominating the local graph. Community-detection methods then provide one possible grouping. Louvain alternates local moves that improve modularity with aggregation into a smaller graph; label-propagation variants update labels using neighboring labels and a specified update and tie-breaking rule. Results can depend on randomness and resolution, and runtime depends on graph size and implementation. Evaluate those properties before choosing a refresh schedule. Once groups are defined, the number of groups spanned by shared contacts becomes a candidate feature to test. NetworkX documents the Louvain procedure.

When it works, a circle looks like what you would expect (the people from one employer, the ones from a university, the relatives), with few edges crossing between them. Two failure cases are worth volunteering. A member whose contacts all know each other, a single dense blob, yields one circle, so every pair scores 1 and there is no signal at all. And ego networks below roughly twenty contacts partition unstably. Run label propagation twice with different random orders and the circles change, so the score moves between nightly runs for reasons that have nothing to do with the member. Fall back to behavioural evidence for both groups.

In the tables above, the circle labels are hand-written. That is the demo standing in for the hard step. Inferring circles is the part that takes the work; scoring a pair once the circles exist is the easy line.

Interview tip: When an interviewer gives you a graph problem, propose the naive overlap metric, then immediately say how it fails, then give the fix. Showing the failure mode is worth more than arriving at the fix directly.

Behavioural evidence outranks structural evidence

Structure tells you who two people know. Behaviour tells you what they do with each other, and when both exist, behaviour wins. The principle underneath recurs in the prediction section: a signal carrying a timestamp of intent beats a signal describing a stable state.

For Wren, in rough order of usefulness:

• Reciprocity of initiation. Close pairs both start conversations; one-directional threads are service relationships.

• Response latency at unusual hours. Answering at 23:40 is a different relationship from answering at 10:15.

• Photos where both faces appear, and events both attended.

• Message register. Acquaintance messages are formal and templated. Close pairs write shorter, sloppier, funnier ones, and a typo-rate or perplexity feature separates them surprisingly well.

• Threads that end with a plan rather than with information.

Assemble these into one vector per pair and score with cosine distance or a small learned ranker. What earns credit is that you separated definition, evidence, and distance function, and did not open by saying cosine similarity.

What a similarity score is allowed to do

State the product surface, because different surfaces impose different requirements on the same score.

The third row catches candidates, and in a subtler way than the usual version of this rule allows. A threshold needs a labelled eval set to pick the operating point, plus scores comparable enough across members that one cut gives everybody a sensible group size. Calibration is the cleanest route to that, and it is required once the threshold must be quoted as a probability, traded against a cost, or reused after a refresh without retuning. It is not required merely because a threshold exists: an ordinal score cut at a tuned percentile is a legitimate first version. Anything user-visible and named needs a defensible cut and a graceful way to be wrong.

---

Should we build it: two gates and a cannibalisation check

Ember, the photo community, has a PM who wants a Cheer button next to the existing Like. "Should we add it" gets answered badly in two ways: yes because it sounds nice, no because it sounds redundant. Both skip the work.

Gate one: if it worked perfectly, would it matter

Name the one metric the feature is supposed to move and ask whether a wild success on it would be genuinely good. If you cannot name the metric, stop. A feature with no metric is not a data science conversation.

For Ember, engagement is actions per active member per week, an action being a post, comment, like, save, or upload. Cheer moves that through two mechanisms: cheers are themselves actions, and a richer reaction vocabulary makes posting feel more rewarding, which raises supply. Attack the first mechanism yourself before the interviewer does. If members substitute a Cheer for a Like they would have given anyway, total actions do not move. Cheer clears gate one only if it adds actions that would not otherwise have happened, or raises posting supply. Adding actions on its own does not count.

Gate two: find demand in today's data

This gate separates the answers. You need a behavioural proxy for demand drawn from the current product, and the strongest proxies are always workarounds.

For Cheer, the proxy is comments. Classify recent comments by intent and measure the share that are pure congratulation with no informational content: "amazing", "so proud of you", strings of celebratory emoji. If that share is 19 percent, a large group of members is paying a high price, opening a keyboard and typing, to express what one tap could express. That is demand, already paid for, in your own logs.

The kill column matters more than the justify column. Any proposal can find a supporting number. Naming in advance the number that would make you drop the idea is what reads as senior, and it stops the analysis from turning into advocacy.

The cannibalisation check nobody volunteers

Most strong candidates offer two gates. The third is what lifts an answer above them.

Every new control competes for the same tap. If Cheer takes 40 percent of the taps that would have been Likes, total reactions barely move and you have added interface complexity for nothing. Worse, if the feed ranker weights Likes and you have just split that signal in two, ranking quality can degrade while raw actions rise.

So: name what the feature takes from, and the metric that reveals the theft. For Cheer that is Likes per post among posts eligible for both controls, with total reactions per post as the guardrail rather than total Cheers.

concept flow

Triage for should we build it

1. 1
   Name the metric

   one number the feature is meant to move, stated with population and window

2. 2
   Success test

   if adoption were unrealistically high, would that number genuinely improve, or just move sideways

3. 3
   Demand proxy

   the workaround in today's logs that shows people already want the outcome

4. 4
   Kill number

   the value of that proxy at which you would drop the idea

5. 5
   Cannibalisation

   which existing behaviour this competes with, and the metric that reveals the trade

6. 6
   Decision

   not ship, not skip, but test, with the guardrail you just named

Interview tip: Passing all three gates never earns the word ship. It earns the word test. Saying "so we should build it" after a clean analysis is the fastest way to lose points you already won.

Why "test it" is the only correct ending

Evidence of demand and a plausible mechanism do not establish the effect of a particular implementation. An appropriately designed controlled experiment can estimate its effect over the tested window and population. Explain which outcomes and adverse effects the test measures, how long they need to mature, and what uncertainty will remain about a broader rollout.

Keep one exception ready: when a change is small, reversible, and a clean test costs more than the information is worth, a monitored rollout to a small percentage is defensible. Offer it only if you can say why this feature qualifies.

---

Predicting user behaviour: label, panel, leakage, action

The fourth costume. Trailhead wants to know who is about to change roles, and churn, propensity to upgrade, and likelihood to file a support ticket all use the same machinery.

Write the label as a full sentence

Almost every weak answer here is weak because the label was never pinned down. Write one sentence containing four things: event, horizon, population, snapshot moment.

"For every member active in month M, predict the probability that their profile shows a new employer in month M plus one."

Then argue with your own sentence, because the interviewer will.

• Event. Is a title change at the same employer a move. Probably not, but a promotion shares many of the same behavioural precursors, so it deserves a separate label.

• Horizon. One month keeps the base rate at 2.7 percent and makes precision hard. Three months raises the rate and blurs the timing. Match the horizon to how long the intervention takes to work.

• Population. Members last seen 14 months ago are unreachable. Scoring them inflates the denominator and wastes capacity.

• Snapshot. The moment features are frozen. This is the one that causes production incidents.

One more argument belongs here, and it is the one that decides who pays for the model. A job change is not churn from Trailhead. A member who moves employers keeps using the professional network, and arguably uses it harder, so the event is a core product moment rather than a loss. That matters because the roll-out calculation later in this lesson attaches a retention value to preventing a move, and Trailhead cannot collect that value. Whoever pays to prevent the move is the member's employer. So read the economics section as an employer's retention team, buying a score that Trailhead sells or that the employer builds on its own HR data. The population, the cost per contact, and the value of a retained employee all belong to the employer. If you would rather keep everything inside Trailhead, change the label to genuine product churn: the member stops opening Trailhead in month M plus one. The same machinery then runs unchanged.

The panel and the embargo

Build the panel by stacking snapshots: for each member and month, take features observable up to the end of that month and attach the following month's outcome. Split by time, never at random, and leave a gap.

The 0.661 in that last row is the test AUC of a gradient-boosted tree fitted on those six columns, and it is worth defining once because this lesson reports it nine more times. AUC is the probability that, if you draw one member-month who moved and one who did not, the model gives the mover the higher score. Coin-flip ordering wins that comparison half the time, so 0.5 is the floor and a model below it is ordering backwards. Perfect ordering is 1.0. At 0.661 the model wins that pairwise comparison about two thirds of the time. AUC says nothing about whether the numbers are the right size, only about the order. A model that scores every probability at one tenth its true value has the same AUC as the correctly scaled version.

Skipping month 19 is deliberate, and the usual justification for it is wrong in a way worth getting right. It is not that a month-18 feature and a month-19 outcome overlap: pairing them is simply how a panel row is built. The actual problem is that the last training row's label lives in month 19. Train on months 1 through 18 and you have already told the model what happened during month 19. Score a member on month-19 features and you are testing on a month the model has partly seen. Skipping to month 20 removes that overlap.

The embargo bites much harder once your features use overlapping windows, which the checklist at the end of this section recommends. With 90-day windows, a month-19 feature is built from events in months 17, 18, and 19, so it shares two thirds of its raw events with the training period. One embargo month is no longer enough. The gap has to be at least as long as your longest feature window, or the same events sit on both sides of the split.

The comparison with a random split is instructive, though not in the direction people usually quote it. Splitting these rows at random gives 0.656 against the time split's 0.661. Resist reading a direction into that. Repeat the random split with eight different shuffles and the AUC lands anywhere from 0.652 to 0.673. That is a spread four times wider than the 0.005 gap itself, and two of the eight beat the time split. On this generator there is nothing for the time split to protect against beyond a clean seasonal term, so the two should agree, and they do. On a real panel, where the product, the logging, and the market all shift, the gap can be far larger, and it usually runs in the flattering direction. But you cannot demonstrate that here. Do not claim a 0.005 difference as evidence of it.

Leakage is a column, not a concept

The abstract version of leakage is useless in an interview. The concrete version is a column whose value was written during or after the outcome window.

Suppose someone adds inmail_opened_outcome_month, a flag for opening a recruiter message during the month you are predicting. Validation AUC jumps from 0.661 to 0.795. That looks like a breakthrough and is worth nothing: at scoring time, on the last day of month M, the column is empty for everyone. The model leans on a feature that is all zeros in production, and live performance lands below the version without it.

The audit is boring and mandatory. For every feature, name the timestamp of the last event contributing to it and assert it falls at or before the snapshot boundary. If a column comes from a nightly aggregate, check when that table is written, not when its events happened. An aggregate running at 03:00 on the first of the month has quietly eaten a day of the future.

Interview tip: When asked how you would prevent leakage, do not define it. Name a specific column in the problem you were just given that would leak, say who would have added it and why it looks reasonable, and give the assertion that catches it.

That tip is not a ban on definitions. The definition is the easy half and everyone has it. Naming the column proves you know where leaks come from: not from malice, but from a well-meaning engineer joining a table that happens to be written a day late. The assertion is what makes the knowledge survive you leaving the team.

Which features carry the signal

Split the same model three ways and the answer is blunt.

Read that table as a demonstration, not as proof. This panel was built so that behaviour carries the signal. One hidden per-member trait sets both the four behaviour counts and the move probability, and the only window onto that trait is the counts. The two columns in the last row are months_in_role, which scores 0.511 alone, and sector_openings_index, which scores 0.530. So almost the whole 0.531 is the market column, and the profile column contributes essentially nothing. The table recovers an assumption rather than discovering a fact, and you should say so out loud.

The reason to believe the same ordering on real data is separate, and it is an argument about time rather than about this generator. A profile attribute is a description of a slowly varying state. That a member is a senior engineer in logistics with four years of tenure is true this month, was true last month, and will be true next month. So it cannot tell you which month anything happens. A behaviour count is a dated event. That they edited their headline twice and answered three recruiter messages in the last 30 days is a fact about a window, and it changes when the member changes. Only a dated signal can locate an event in time, and a monthly prediction is entirely a question of timing. Profile attributes set the base rate; behaviour moves you off it.

The design consequence: if your pipeline holds 40 profile attributes and 4 behavioural counts, you built it backwards, and the fix is not more attributes. It is more behavioural windows, the same counts at 7, 30, and 90 days plus their ratios, which encode acceleration. A member whose 7-day search count is triple their 90-day average is in a different state from one at a steady simmer, and a flat count cannot say so.

Rank metrics, not accuracy

At a 2.7 percent base rate, a model predicting that nobody moves is 97.3 percent accurate. It is also worthless: it never puts anybody in the queue, so it never causes an action. That is the whole objection to accuracy on a rare event. Accuracy rewards the majority class, and the majority class is the one you do not care about. The metrics that matter follow instead from what the output is for.

Bucketing the same scores into deciles and reading the top four:

The lift column in the first table needs a definition before it carries any weight. Lift is the event rate inside the group you selected divided by the event rate in the whole scored population. A lift of 1 means your selection is no better than picking at random. A lift of 3.5 means the members you picked move three and a half times as often as a random member-month. It is a ratio, so it says nothing about volume: six people with a perfect hit rate have a spectacular lift and fill nobody's queue.

Those are two different claims. The first says ordering works: the top 500 member-months move at 9.8 percent against a 2.8 percent base, a lift of 3.5. That lift falls as you go deeper, to 3.0 at 2,000 and 2.0 at 10,000, which is exactly what a working ranker looks like: the best rows come first. Lift over the whole top decile, 4,500 rows, is 2.5. Quote the depth whenever you quote a lift, because the number is meaningless without it.

The second says the probabilities mean something. Calibration compares what the model predicted against what actually happened, bucket by bucket. The top decile predicts 6.6 percent and observes 6.9 percent, close enough to use in an expected-value calculation rather than only as a sort key. Bucket 8 is the one visibly off: it predicts 4.0 percent and observes 4.8 percent, under-predicting by a fifth.

Which one matters more depends on the intervention. For a team working a fixed-length queue, ordering is everything and calibration is a nicety. For a decision about whether contacting someone is worth the cost, calibration is everything, because that decision compares a probability to a price.

The number that actually decides the roll-out

This is the calculation almost nobody produces unprompted, and the one that turns a model into a recommendation. Say the employer's retention team sends a personalised outreach for 4.20 in currency per contact. That outreach persuades 11 percent of would-be movers to stay, and a retained employee is worth 520 over the following year.

Cut the ranked list at five depths and the arithmetic reads:

Read the units before the curve. The scored set is months 20 through 24, five snapshots of 9,000 members, so the contact count is member-months pooled across the whole window, not contacts in one month. Every row is a five-month total.

Change the selection rule and the arithmetic changes with it. Every cut in that table is global, taking the best rows anywhere in the five months. Take the top 2,000 within each month instead, the same 10,000 contacts under a fixed monthly quota. You pick up 555 movers rather than 563, for a net of minus 10,254 against the global cut's minus 9,796. Slightly worse, and the reason is the point. A fixed monthly depth forces you to contact weak candidates in the cold months, in order to leave good ones uncontacted in the hot one.

Net value peaks near 2,000 and turns negative before 5,000. But where it peaks, where the next contact stops paying, and where the whole campaign goes under water are three questions, and a five-row table answers none exactly. Sweeping every depth, and adding the rule the calibration paragraph set up, gives three cut points.

The top 2,000 also do not spread evenly across the five scoring months:

Three numbers, three meanings. The 1,902 is the marginal stopping point: net peaks there at about 1,450, so contact 1,903 and you are worse off than stopping. The 3,254 is where cumulative net crosses back through zero, the point where the programme as a whole stops paying rather than where the next contact does. Sliding between those two is how this calculation usually gets misquoted. The 1,042 is what the principled rule picks: contact a member only when p times 0.11 times 520 beats 4.20, so only when p clears 7.3 percent.

Do not over-read the gap between 1,902 and 1,042. Both are read off a step function built from 45,000 coin flips, and neither is stable. Bootstrap the scored rows 200 times, re-sorting and recomputing the curve each time. The marginal stopping point then lands anywhere from about 1,000 to about 2,100, and the cumulative break-even anywhere from about 2,100 to about 5,400. The first of those intervals contains 1,042, so at this sample size the two rules cannot be told apart. The curve itself is nearly flat through the region: net value stays within 200 of its peak for every k between 1,267 and 2,044. Report ranges, not four-significant-figure stopping points, and say the two rules are indistinguishable here rather than that they disagree. Only perfect calibration makes the two rules coincide.

One local observation is worth reporting honestly rather than dressing up. The 860 member-months ranked between the two cutoffs predict 0.067 on average and actually move at 0.080, so they pay despite failing the calibrated bar. That looks like the under-prediction bucket 8 showed, but it is not: all 860 of those rows sit inside bucket 9, whose aggregate calibration is good (0.0663 predicted against 0.0689 observed). This is a narrow slice with 69 movers against about 58 expected, roughly one and a half standard errors. Interesting, not established.

All of this turns "we built a model with AUC 0.66" into something an operator can act on. The recommendation reads: "contact the top 2,000 member-months across the five-month window, not a flat 400 a month. That is between 262 and 295 in each of the four ordinary months and 881 in month 24. It buys roughly 19 retained members and 1,270 in net value over the window. The next contact stops paying somewhere around 1,000 to 2,100, and the campaign as a whole goes under water somewhere past 2,100 to 5,400."

Reconciling capacity and economics

Three different answers to "how deep should the queue go" have now appeared, and you need them in the right order rather than picking whichever one is nearest.

Economics sets the depth you want: the net value curve, about 1,900 contacts by the marginal rule and about 1,000 by the calibrated one, indistinguishable at this sample size. Capacity sets the depth you can staff. Put both in the same units before you compare them, which is the step the units warning above exists for. The 1,900 is member-months pooled over five months, so it is about 380 a month, a dozen a day. This lesson never states the employer's staffing, and the 400 a day it opened with belongs to a different illustration, but a team that size would work all 45,000 scored member-months in 113 days, well inside the window. Any capacity near that is more than thirty times the economic optimum, so here economics binds and capacity does not.

That is the case worth understanding, because the temptation runs the other way. With people free you still work to the economic optimum and redeploy the slack, and the table above prices the alternative: contacting 10,000 rather than 2,000 turns a net of 1,267 into minus 9,796. Capacity binds only when the optimum sits above what you can staff, and then precision at that queue length is the right metric, because the depth is fixed by something outside the model.

And because the optimum moves with the hiring season, with 881 of the top 2,000 landing in one month, recompute the depth every month rather than fixing it once.

Where to push next

Net value is linear in the uplift rate, the retained value, and the cost per touch, and two of those three are yours to move. Doubling the persuasiveness of the outreach lifts the optimum from about 1,900 contacts to about 9,600 and takes net value from 1,450 to roughly 23,000. Halving the cost per touch, from 4.20 to 2.10, moves the optimum the same way and takes net value to roughly 11,600. Either lever is worth many times more than two more AUC points. Cost per touch is the one a retention team controls most directly: channel, template, and how much human time each contact consumes. The retained value is set by the business and is not yours. Pushing on the economics rather than on the model is the product judgement being tested.

checklist

Before a propensity model goes live

• Label sentence: event, horizon, population, and snapshot moment written in one line

• Time split: an embargo period between train and test, never a random split on a panel

• Feature audit: last contributing timestamp at or before the snapshot for every column

• Behaviour windows: the same counts at 7, 30, and 90 days plus their ratios, not one flat count

• Rank metric matched to the intervention: precision at the queue length the team can serve

• Calibration: top-decile predicted against observed, required if any decision compares a probability to a cost

• Net value curve: value against contact volume, with the marginal stopping point and the cumulative break-even named as two separate numbers

• Retraining: a fixed cadence plus a drift monitor that can trigger it early

The embargo in bullet two has to be at least as long as the longest window in bullet four, or the two bullets contradict each other.

Churn models and the intervention trap

A churn model ranks accounts by probability of leaving, the retention team works the top of the list, and six months later the campaign reports a healthy save rate. The problem is that the highest-probability churners are often the least persuadable: the contract ended, the person left, the use case disappeared. Members sitting at a 30 percent probability with genuine ambivalence are the ones a check-in could flip, and they are at rank 12,000.

What you want to rank by is the change in probability the intervention causes, which is an uplift model. Start by holding out 10 percent of the targeted top and contacting nobody in it. That tells you whether the campaign is incremental at all, or whether it is a well-sorted list of people who were staying anyway.

Then say what that experiment cannot do, because this is where a candidate should notice rather than overclaim. Treatment varies only inside the top, so it produces no treated outcomes anywhere near rank 12,000. An uplift model fitted on it has no support exactly where the persuadable members sit. It reads flat everywhere it can see, so extrapolating predicts nothing at rank 12,000, the opposite of the truth.

Uplift labels need randomised treatment across the score range: a few percent of contacts spent at random, stratified over deciles. Those contacts look wasteful on the net value curve, and that is the price of learning influenceability rather than likelihood. Size that budget by the uplift you need to detect, not by what looks affordable. Uplift is a difference of two noisy rates, so its standard error is larger than either rate's. A token 1 percent returns a flat zero everywhere, and that gets mistaken for an answer. Whenever a model output drives an action, ask whether you are ranking likelihood or influenceability.

---

The whole family in ninety seconds

Compressed, to rehearse the shape:

"For a feature question I would find outcomes members already reach through effort. I would measure the traffic each one carries and how many steps it costs, rank by volume times steps, then divide by build cost. On Wren that puts live location first, because picking a restaurant scores higher on friction but leaves the app. For similarity I would state what closeness means for the product. Then I would note that shared-connection counts, Jaccard and Adamic-Adar are all counting the same overlap, and all pick the coworker. The fix is to score how many distinct circles the shared contacts span, plus behavioural evidence like reciprocal initiation and late-night response latency. For should-we-build I would check that a wild success moves a named metric, then find the workaround in today's logs. I would say in advance the number that would kill the idea, name what it cannibalises, and conclude that we test rather than ship. For prediction I would write the label as one sentence with event, horizon, population, and snapshot. I would split by time with an embargo at least as long as my longest feature window, and audit every feature's last timestamp. I would lean on behavioural windows over profile attributes. And I would finish with a net value curve, naming both where the next contact stops paying and where the campaign as a whole does. Then I would check that number against what the team can actually staff."

Every clause is checkable, and none of it required a product opinion.

---

Common traps

Answering a feature question as a visionary. Fix: open with "here is the behaviour in the logs I would look for". Only name a feature after you have named the friction it removes.

Citing a competitor as evidence of demand. Their user mix, price point, and business model differ from yours. Fix: name the internal signal that would confirm the same demand, and go measure it.

Ranking ideas by friction alone. The highest-friction job often leaves your app entirely and needs a partnership. Fix: score build cost on a stated scale and divide by it before choosing what to ship first.

Reaching for cosine similarity before defining closeness. Fix: one sentence on what closeness means for this product, then the features, then the distance.

Trusting shared-connection counts. They rank the coworker above the closest friend, and normalising the count does not help, because Jaccard and Adamic-Adar are still counting overlap. Fix: score how many distinct circles the shared contacts span, and add behavioural evidence.

Assuming a user-visible score has to be calibrated. A named close-friends group needs a threshold. A threshold needs a labelled eval set to pick the operating point, plus scores that mean the same thing across members. Fix: decide the surface first, then say how you would choose and re-check the cut, calibrating when it must survive a refresh or be traded against a cost.

Concluding "so we should build it". Fix: the analysis earns a test, never a launch. Say the word test.

Skipping cannibalisation. A new control takes taps from an old one and can split a ranking signal in two. Fix: name what it competes with and the guardrail that reveals the trade.

Splitting a panel at random, or adding a feature written during the outcome window. Fix: split by time. The embargo must be at least as long as your longest feature window. Then audit the last contributing timestamp for every column, nightly aggregates included.

Reporting accuracy on a 2.7 percent base rate, or building a model with no named consumer. Fix: report precision at the queue length the team can serve. Report lift over base too, and always quote the depth with the lift. And before modelling, finish the sentence "this ranked list goes to X, who can act on Y per day".

Quoting a stopping point to four significant figures. A net value curve built from a rare label is nearly flat near its peak and moves under resampling. Fix: bootstrap it and report a range.

Ranking churn probability when you meant persuadability. Fix: hold out 10 percent of the targeted top, to measure whether the campaign is incremental at all. Then spend a small randomised exploration budget across the whole score range for uplift labels. The top-of-list holdout on its own gives you no treated outcomes where the persuadable members are.

---

Quick self-check