1.4 Short-Term Wins Versus Long-Term Health

A change can improve this week's metric while creating a cost that appears later. Heavier ad load, more notifications, an aggressive first-order discount, and ranking by clicks are examples to examine. The immediate gain may be real, but a short test may miss a later effect on retention or value. This lesson explains how to identify that trade-off, estimate the delayed cost, and distinguish measured outcomes from assumptions about the future. You will also evaluate when an expected longer-term benefit could justify a loss over the first quarter.

Why this matters in interviews

The interviewer almost never asks "is this a short-term versus long-term tradeoff." They hand you a prompt where the tradeoff is buried and watch whether you find it. Three real openers:

• "We ran a test that increased ads per session from three to four. Revenue per user went up eleven percent, the result was significant, retention showed no significant difference. Do we ship it?"

• "Growth wants to double push notification volume. What would you want to see before you agree?"

• "One advertiser has by far the best click rate on our network. How would you tell whether that is good news?"

Every one has the same skeleton: a lever raises yield per unit of exposure and, on a delay, shrinks the base of exposures. The candidate who says "revenue is up eleven percent and retention is flat, ship it" has answered a question about a two week window nobody cares about. Contrast the candidate who says: "flat retention over two weeks is what a retention effect looks like before it shows up. Here is how long I would need and what I would watch instead." That candidate is doing the job.

There are four scoreable moves, and they are what the rest of this lesson drills:

1. Name the borrowed resource. What exactly is being spent, and does it regenerate.

2. Convert both sides into one currency, which is almost always discounted contribution per user over a stated horizon. Contribution is revenue minus the variable cost of serving that user. Discounted means every future dollar is pulled back to today's value at the company's weekly cost of capital. The horizon is a number of weeks you fix before you look at the result. Sections below build each piece for a real decision.

3. Find something observable inside the test window that predicts the part you cannot wait for.

4. Commit, in advance, to a decision rule and a long-run measurement that will still be running after everyone has moved on.

One word in the tip below carries most of the weight, so define it now. A censored outcome exists but has not been observed yet. Follow users for two weeks and ask about twelve-week retention. Nobody has reached week twelve, so there is no measurement at all, not even a noisy one. That is categorically different from a null result, where you measured and found nothing. A null result can be sharpened by adding users. A censored one cannot: every user you add arrives with the same unobserved outcome, so you have more rows and still zero measurements. That is why "no sample size fixes it" is a statement about the calendar, not the budget.

Interview tip: When a prompt reports a positive result with "and retention was flat," treat that as the interesting sentence, not the reassuring one: ask first how long users were followed, because a retention number nobody waited for is censored rather than null, and only then what effect size the test could actually resolve.

---

The shape of the trade

Start by writing the metric as an identity. For an ad-supported product:

weekly revenue = active users x sessions per active user x ads per session x click rate x revenue per click

Raising ads per session multiplies one term by 4/3. Nothing in the identity says the other terms hold still. In practice two of them respond, on different clocks:

• Click rate drops almost immediately, because the fourth ad in a session is seen with less attention than the third. This shows up inside days.

• Active users and sessions per active user drop slowly, because annoyance accumulates and quitting an app is a decision people make once, at a moment that has nothing to do with the week the change shipped.

That gap in clocks is the whole problem: the fast terms are the ones the change was designed to move, and the slow ones are what pays for it.

Renewable versus non-renewable levers

Not every borrow is equally dangerous, and this distinction is the fastest way to sound senior. Some levers deplete something that grows back if you stop. Others spend something you cannot get back at any price.

The right-hand pattern matters more than the exact numbers. When the borrowed resource does not regenerate, a reversible experiment is a lie: you can turn the treatment off, but users who revoked notification permission do not come back.

The phrase one-way door gets used loosely, so pin it down: a one-way door is a change whose cost survives the rollback. Reverting the treatment flag restores the code, not the permission. The operating-system notification toggle a user flipped off is still off, and getting it back requires them to walk into system settings on your behalf, which almost nobody does.

That changes how you have to work. The default safety net for a risky launch is ship, watch, revert, and it works only because reverting undoes the damage. Behind a one-way door that net does not exist, so you have to gather the evidence before launch. That leaves two options: a small pilot sized so the permanent damage stays inside a population you are willing to lose, or a proxy on declining behaviour that never exposes a large group at all. "Ship to ten percent and monitor" is a fine plan for ad load and a bad one for notification permission. The difference is arithmetic rather than caution: a revert can give back ad load and cannot give back a permission.

Interview tip: Say the words "this is a one-way door for notification permission, so I want the decision made before the launch, not after" and you have separated yourself from every candidate who proposed shipping and monitoring.

The one-line diagnostic

For any prompt, ask which large term in the identity the change does not touch directly, and by what mechanism it responds. Name a mechanism and you have a hypothesis worth measuring. Fail to name one and the change is probably a genuine free win, which you should say rather than manufacture a concern. Reflexively finding a dark side in every idea reads as contrarian, not rigorous.

---

A dataset to make this concrete

The running example is Nightjar, a mobile long-form audio app monetized with in-stream audio ads. The product team ran a 13 week experiment that raised ad load from three to four spots per session, on 40,000 users split roughly evenly between the arms.

The analysis below runs on a panel frame where one row is one user in one week.

Three things are true of this panel by construction. Hold them as the ground truth your reading procedure has to recover. The treated arm gets a third more inventory. Its click rate per spot is slightly worse, 1.85 percent against 2.10 percent, because the fourth slot draws less attention. And its churn hazard starts identical to control, then drifts upward with calendar time. The weekly hazard is 3.0 percent in control, and 3.0 percent plus 0.18 points per elapsed week in the treated arm. One more number carries every dollar figure below: sessions are drawn with the same mean in both arms, 4.14 per active user-week. Multiply it by the spots per session, the click rate and the 0.42 USD per click for either arm's mechanical revenue per active user-week.

The hazard rule is a stand-in for accumulating annoyance, not a per-user exposure count. A treated user who opened the app twice all quarter carries exactly the same rising hazard as one who sat through sixty ads. Nobody hands you a dataset with the mechanism labelled; the point of knowing it here is to check whether your reading procedure would have caught it.

---

Reading the experiment honestly

The naive read

Compute revenue per assigned user and survival per arm. Always divide by users assigned, never by users still active, or you have conditioned on an outcome the treatment causes. The readout is below: weekly revenue per assigned user in each arm, the lift between them, and the share of assigned users still active in each.

If your team called the test at two weeks, which is what most teams do, the readout was a 19 percent revenue lift with no visible retention effect. Ship it and write the doc.

The read that changes your mind

Two things in that table should stop you. The lift is trending down, from 18 percent in week one to 4 percent in week thirteen, and the survival curves have separated by ten points. Note the word trending: on 20,000 users per arm the weekly lift rises in five of its twelve transitions, including 4.9 points from week five to week six. The path underneath is monotone, 17.5 percent falling to 1.5 percent on the generator's own parameters. But no thirteen week readout at this size marches down in a straight line, and narrating the wiggles is reading noise.

After a full quarter, the cumulative revenue readout is still positive: plus 12.7 percent. There is no week in the test where the treated arm's weekly revenue falls below control. Every dashboard in the company says this worked. And it is still only barely positive a full year out from the test's start, as the next section shows. It is a wash somewhere around fourteen or fifteen months, and clearly destructive beyond that.

The failure mode in this class of change is not that the test turns negative and someone ignores it. It is that the test never turns negative inside any window a company will wait. The loss is carried by a shrinking base, and that base's absence compounds after the experiment is shut off.

Interview tip: State explicitly that a decaying treatment effect and a widening survival gap are the same phenomenon seen twice, because revenue per assigned user equals revenue per active user times survival, and here the per-active ratio wanders in a narrow band between 1.14 and 1.21 and never falls, while the survival ratio drops from 1.00 to 0.857.

Do that decomposition arithmetically, and keep both sides on the same footing. In the observed sample, week thirteen's per-active ratio is 1.2145 and its survival ratio is 0.8570. Multiply them and you get 1.0408, the 4.08 percent week-13 lift the table prints. The generator's own parameters give the matching pair: a mechanical per-active ratio of 1.1746 and a true survival ratio of 0.8643. Their product is 1.0152, the 1.5 percent path mentioned above. Use one pair or the other. Multiplying a truth-side numerator by a sample-side denominator produces a number that appears in neither.

Compute that decomposition rather than asserting it, because the tempting explanation is a different one and it is wrong here. The tempting story is compositional: the treated arm sheds its least tolerant users first, so the survivors are a tougher crowd. Check the direction before you say it. If that were happening, the treated survivors would get steadily more tolerant, and revenue per active user in the treated arm would drift up. The effect would then offset the decay rather than cause it.

Nightjar has no such heterogeneity. Every treated user carries the same hazard of 0.030 plus 0.0018 times the week, so any drift in the per-active ratio is sampling noise. The panel bears that out on both arms at once. Look at each arm's revenue per active user separately, because a ratio can move for two reasons and you need to know which. The heavy arm runs 0.1302 in week one and 0.1284 in week thirteen, a 1.4 percent slide, against a mechanical level of 0.1287 USD per active user-week. The control arm runs 0.1103 down to 0.1057, a 4.2 percent slide, against a mechanical 0.1095. So the week-13 ratio of 1.2145 sits above the mechanical 1.1746 mostly because the control arm's last week drew low, not because treated survivors toughened up. This is the moment to say: "that pattern is consistent with selective attrition but does not demonstrate it, and here is the plot that separates them." Say that rather than reaching for the more interesting mechanism. The plot is revenue per active user by arm: flat means no compositional shift, a rising treated line means selective attrition. Use it to decompose only, because the number you report still divides by assigned users.

---

Lifetime value as the arbiter

You cannot compare "plus 12.7 percent revenue this quarter" against "ten fewer retained users per hundred" until both are in one currency. That currency is lifetime value.

Define it so two people compute the same number

Define lifetime value before using it to compare alternatives. Specify the population, value measure, observation window, projection assumptions, and treatment of future cash flows so another analyst can reproduce the calculation.

For Nightjar the variable cost is content licensing plus delivery, roughly 0.045 USD per active user-week. Ad revenue is high margin but not free.

A single exponent moves every number in this lesson. Survival from week 9 to week 13 is the product of four weekly survival factors, one per interval. So the constant weekly hazard consistent with that stretch is 1 - (S13 / S9) ** (1/4): five observed points, four intervals, hence the quarter power. Fit it on the tail rather than the whole curve. The early weeks of any experiment carry startup effects and the pre-treatment regime. You want the hazard the product is running at now, not its average since launch.

So the test window banked an extra 0.174 USD of contribution per assigned user, and left the treated arm with a weekly churn hazard of 4.96 percent against 3.00 percent. Those two numbers are the whole decision.

Estimating lifetime value: four methods that answer different questions

Interviewers like to ask "how would you estimate it," and the good answer is a question back: estimate it for what. The method follows the use.

For an ad-load decision you want method two, and the reason is specific: the change moved exactly two things, and method two prices them separately. Contribution per active week went up, from 0.0607 to 0.0834. Weekly survival went down, from 97.0 percent to 95.0 percent. Multiplying a survival curve by a per-week contribution keeps those two knobs apart, so you can attack the estimate of either one without redoing the other. The historical-cohort method cannot do that at all, because it reports a realised total for a world in which both moved together. A predictive model can in principle, but it would need labels you do not have yet for this change.

The same structure is method two's weakness. It assumes the hazard you estimated off five tail points holds forever. Nightjar's treated hazard is still rising at 0.18 points a week, so the flat-hazard version is the optimistic bound on the damage. That admission belongs in your answer, not in a footnote. For the Portside question below you want method three.

The first-session model, at Portside Goods

Switch products for a moment, because interviewers love this variant. Portside Goods is a home-textiles marketplace. The question: a visitor lands on the homepage for the first time and has not bought anything. What is that visitor worth?

Frame it as supervised regression on a horizon-bounded target: discounted contribution over the 12 months after the first visit. Train on visitors whose first visit was at least 13 months ago, so every label is complete. Use only features available at prediction time. That means acquisition channel, the query that produced the click and its commercial-intent bucket, brand against category ad, and device class and operating system. It also means browser locale as a language proxy, country and metro, and calendar week. Calendar week carries real signal, because a first visit in mid-November is worth far more than the same visit in February.

Two structural points that separate a good answer from a recital:

Model the zero. Most first-time visitors never buy, so the target is zero about 92 percent of the time. The problem this creates is not bias in the average, because a squared-error regression estimates the conditional mean whatever the shape of the target. The problem is resolution. With 92 percent zeros, almost all of the loss is generated by the zero mass. So the fit spends its capacity separating zeros from zeros, and has very little left to distinguish a visitor worth 20 USD from one worth 200. Leaf-level means among the 8 percent who matter end up estimated on a handful of rows and bounce around from one refit to the next.

Splitting the problem fixes both halves at once. A classifier for "buys within 12 months" gets a loss that suits a probability; a regression on positive labels only gets a loss that suits a skewed positive amount. You also get a diagnostic one model cannot give you: when the expected value is wrong, you can see whether the buy rate or the basket size is miscalibrated. Different bugs, different fixes. Either use that two-part model, or use a loss designed for non-negative skewed targets such as Tweedie. State which and why.

Predict the segment, not the person. The output is almost never used per user. It is used to set an acquisition bid, a welcome-offer size, or a channel budget, all of which are decisions made over groups. That means you need calibrated conditional means for segments, not sharp individual ranking.

Do not let that argument push you into fitting a shallow tree as your estimator, because a shallow tree is a poor estimator and a good interface. Fit the boosted ensemble for the numbers, then fit a single depth-3 or depth-4 tree to the ensemble's predictions and bid off that tree's leaves. The tree is the segment map humans argue about and finance signs off on; the ensemble is what produces the value in each leaf. Saying that out loud is what separates a candidate who understands why segments matter from one who has confused the reporting object with the model.

Concretely, the Portside estimator is two fits and one multiplication.

A visitor's expected value is the first row's probability times the second row's prediction.

The loss on the size model is load-bearing, and interviewers catch people here. The decomposition only multiplies through for means: expected value is probability of buying times the mean margin among buyers. So the test any candidate loss has to pass is whether it returns a conditional mean.

A quantile loss such as absolute_error returns the conditional median, and order values are skewed enough that the median sits well below the mean. Take a two-part simulation with 91 percent zeros and a lognormal positive part. There absolute_error recovers 0.86 of the true mean at mild skew and 0.53 at heavy skew. The default squared_error recovers about 1.02, and gamma runs from 0.98 down to 0.88. The bias runs one way and is worst where the tail is fattest, so when dispersion differs across channels the median version can rank channels backwards. On that criterion the numbers pick squared_error, and it is the right default here: it targets the mean directly, which is the quantity the decomposition needs.

gamma is worth reaching for when order sizes span orders of magnitude. Its log link means the model is fitting proportional rather than absolute error. A single 4,000 USD order does not dominate the loss the way it does under squared error. That is a real property, and it is also why gamma drifts to 0.88 above: pulling the fit toward proportional accuracy costs you some fidelity on the mean. Check the recovered mean on your own data before you accept the trade. The one argument that does not work is "unlike poisson it does not force variance to equal the mean." That is true, but poisson was never the competitor. squared_error is. Tweedie is the single-model alternative, fit on every row including the zeros, not a swap for the size model. In scikit-learn that means TweedieRegressor, not HistGradientBoostingRegressor, which has no tweedie loss.

The trap candidates miss is staleness. Your labels are 13 months stale by construction, so the model has learned last year's checkout flow, last year's shipping speeds, and last year's competitor set. Say out loud how you would handle it. The standard fix is to train on complete labels but recalibrate on recent cohorts using an early surrogate, which is exactly the proxy machinery two sections down.

Interview tip: For any lifetime value question, name the horizon and the margin definition in your first two sentences, before any modelling talk, because those two choices move the answer more than the model class ever will.

---

Horizon and discounting decide the answer

Now price the Nightjar decision. Banked contribution is known. Forward contribution is a discounted sum over the surviving base.

You should be able to derive that sum at a whiteboard. A user still active at the end of the test is worth margin next week only if they are still around, which happens with probability 1 - hazard. And that dollar is worth less than a dollar today, which divides it by 1 + d. Roll both shrinkages into one factor f = (1 - hazard) / (1 + d) and week k is worth margin * f ** k. Summing k = 1 to H is a finite geometric series, f * (1 - f ** H) / (1 - f), which is what the horizon table below evaluates. The sum starts at k = 1 rather than k = 0 because week 13's revenue is already counted in banked; starting at zero would bank week 13 twice.

Evaluate that at four horizons and you have the punchline of the lesson.

Read the first column exactly as it is labelled: these are weeks after the test ends, and banked already covers the 13 weeks inside it. So the 52-week row is 65 weeks of total elapsed time, about fifteen months from the day the test started, and the 104-week row is 117 weeks, about twenty-seven months. If your company's standard horizon is twelve months, the row you need is not in the table. It is 39 forward weeks, which gives a total of +0.055 USD per assigned user, positive rather than a coin flip.

Intervals come from the user-level bootstrap at the end of this section. Carry them into the table: they change no label, and they are what makes "coin flip" a measurement rather than a turn of phrase.

The same experiment and the same data give four different answers. Nothing about the evidence changed, only the horizon.

This is why the horizon is not an implementation detail you can leave to the analyst. It is the decision. Choose it before you see the result, and choose it on a principle you can defend. Match it to the payback period the finance team already uses for customer acquisition, or to the planning cycle the decision belongs to. If your company pays back paid acquisition in 9 months, evaluating product changes on a 13 week horizon means product is allowed to destroy value that marketing is not.

Discounting, briefly and correctly

The rate does less work than people expect. At 12 percent annual the weekly rate is about 0.218 percent. Over 52 forward weeks it shaves the forward term by about 3 to 4 percent at our hazards. That is 4.2 percent on the control arm and 3.4 percent on the treated one. With no churn at all it would be 5.6 percent. But survival has already stripped most of the weight out of the far weeks, so discounting finds less left to remove. The higher the hazard, the less the discount rate matters.

Doubling the rate to 24 percent leaves all four verdicts in the table standing, but it does flip the sign of the marginal one. The 52-week row moves from minus 0.010 to plus 0.002, and the break-even point slides from about 50 forward weeks to about 53. A sign change on the exact row the whole argument turns on is worth knowing about. It is the best defence of the "coin flip" label. A quantity that changes sign when you double a rate nobody can pin down is not a verdict. Do not spend interview time arguing 10 against 15, though. Spend it on the horizon and the hazard estimate, which are worth ten times more. The rate matters mainly at a startup, where the honest discount is the return on the next dollar of runway rather than a textbook cost of capital.

Turn it into a break-even, which is the version that survives disagreement

Nobody will agree with your hazard estimate. Rather than defending it, invert the question: how much extra churn would the change have to cause before it stops being worth it. Bisect for that threshold on a 52 week forward horizon and the change breaks even at 0.0189 of extra weekly hazard, 1.89 points, while the data says 0.0196, or 1.96. Now the meeting is about one number that everyone can attack, which is exactly where you want it. If someone believes the true incremental hazard is 1.5 points, they have made a falsifiable claim and you can go measure it.

What you cannot do is stop there and declare the argument over, because a 0.07 point gap between an estimate and a threshold is not a result. Both sides come from the same 40,000 users, so both carry sampling error, and the threshold is the wider one. Resample users with replacement and push each of two thousand replicates through the whole pipeline, re-deriving the hazard, the margin, the banked contribution and the break-even inside every one.

The estimated incremental weekly hazard is 1.96 points, with an interval from 1.74 to 2.19. The break-even estimate is also uncertain: 1.90 points, with an interval from 1.29 to 2.58. Their difference is 0.06 points, with an interval from −0.69 to +0.74. At 52 forward weeks, or 65 weeks from the test’s start, the estimated value change is −0.01 USD per assigned user, with an interval from −0.11 to +0.09. About 41% of the bootstrap replicates have positive value at that horizon. That resampling fraction is not a posterior probability that the true value is positive, and an interval crossing zero does not establish equivalence. It shows uncertainty under the selected sampling and projection assumptions. See SciPy’s bootstrap documentation for the repeated-sampling interpretation of a confidence interval.

That is what "coin flip" in the table above actually means. Now check it against the truth of the simulated world, carefully, because the obvious comparison is the wrong one. The churn rule really does produce 1.890 points of extra tail hazard. But the 1.886 sitting next to it is not a truth-side number. It is the sample break-even quoted above as 0.0189, computed from this sample's week-13 margin, banked contribution and surviving base. Every one of those carries noise. Comparing a generator constant against a sample estimate and calling the near-match a property of the simulated world is exactly the apples-to-oranges the bootstrap exists to prevent.

Do it consistently instead. Compute the break-even from the generator's own parameters and it is 1.485 points, against a true incremental hazard of 1.890. The true 52-week-forward net is minus 0.063 USD per assigned user. The simulated world is a loss, not a tie. What makes this sample look like a tie is essentially one number. The control arm's week-13 contribution drew low, 0.0607 against a true 0.0645, which flatters the treated arm's relative margin. Swap only that one input and the break-even falls from 1.886 to 1.457.

So the closing line is neither "the decision is made" nor "the truth is a coin flip." Our point estimate landed near break-even partly because a single noisy week-13 number flattered the change. The sampling interval is wide enough to cover both the tie we see and the loss underneath it. This is precisely the case for holding a slice back and settling it with evidence rather than with whoever owns the revenue target. This interval sits on top of the flat-hazard extrapolation, not instead of it.

Interview tip: Convert every long-term argument into a break-even threshold on one parameter, because "we need to believe churn rises by less than 1.9 points per week" is arguable and "I think this is bad for users" is not.

---

Proxy metrics: reading a one-year effect in three weeks

You cannot run a 13 week test on every change, and even 13 weeks did not answer this one. So you need short-horizon quantities that predict the long-horizon outcome.

What a proxy actually has to satisfy

Three conditions, and candidates usually name only the first.

1. It correlates with the long-term outcome. Necessary and nearly worthless alone.

2. The treatment's effect on the long-term outcome runs through it. Suppose a change can move the long-term outcome without moving the proxy. Then the proxy reports "no effect" on exactly the changes you most need to catch. This kills most candidate proxies.

3. It has been validated on past experiments where you eventually saw the truth. A proxy is an empirical claim about your product, not a definition. So it must be fit and checked against experiments with known long-run readouts.

Condition two is the surrogacy condition, sometimes named after Prentice, and it is worth being blunt about its status: you cannot check it inside a single experiment. Verifying that the treatment's whole effect flows through the proxy requires the long-run outcome, and the long-run outcome is the thing you do not have. That is not a technicality you can design around. It is why the condition is stated as a requirement rather than as a test.

Condition three is the only operational version of it, which is what makes it the real work. Keep a library of every experiment that ran a long holdback. Regress the long-run effect on the candidate short-run effect across those experiments. Then ask the question that actually tests surrogacy. Conditional on the short-run effect, does the long-run effect still depend on which experiment it came from? If it does, the proxy is not carrying the whole mechanism, and the leftover dependence is the part it misses.

Report prediction uncertainty on the proxy’s intended target. In a simple in-sample decomposition, an R-squared of 0.4 leaves a residual root-mean-square spread about sqrt(1 − 0.4), or 0.77, times the outcome’s root-mean-square spread; degrees-of-freedom conventions can change the exact ratio. R-squared alone cannot tell you whether a new prediction of +2% excludes zero. You also need the outcome scale, prediction design, and out-of-sample error, especially when there are only twenty past experiments.

Proxy candidates for the two canonical levers

Read that right-hand column as hypotheses, not findings. Nothing here has regressed any of them against a long-run readout, so none has cleared condition three on your product. What they have is a plausible mechanism story, which is where a proxy starts, not where it earns the right to arbitrate a launch. Presenting the column as established proxies is exactly the recital this section warns against.

Notice the pattern that generates the column. The good candidates measure whether the user came back, or whether the user actively declined the thing you added, never the interaction you increased. The declines are the strongest kind, because they are cheap to detect and near-impossible to explain away. Revoking notification permission is an unambiguous week-one vote about a cost that will not reach revenue for months.

Detecting the creatives that poison the surface

A related question comes up constantly in ad-network interviews: without labels, find the ads that win clicks by degrading the surface they run on. The measurement idea is to look at the same users afterwards, on other inventory.

For each creative, take the users who clicked it. Compare their click rate on everything else in the two weeks before that click to their click rate on everything else in the two weeks after. A creative that delivers what it promises leaves that number unchanged. A creative that misleads leaves the user less willing to trust the whole surface, and their subsequent click rate on unrelated ads drops.

One design decision inside the query is easy to get wrong, and this lesson's own headline rule catches it. A user who was driven off the surface entirely has no impressions in the after-window, so an average over that window returns NULL. Filter those users out and you have dropped exactly the people the bad creative damaged most. That is conditioning on an outcome the creative causes, and it biases trust_delta toward zero precisely on the worst offenders. So require a before-window, which is pre-exposure and legitimate, but keep the disappearances in. Count them, and treat a user with zero subsequent impressions as the strongest form of not clicking.

WITH first_click AS (
  SELECT user_id, creative_id, MIN(clicked_at) AS t0
  FROM ad_click
  WHERE clicked_at BETWEEN CURRENT_DATE - 90 AND CURRENT_DATE - 21
  GROUP BY 1, 2
),
windows AS (
  SELECT f.creative_id,
         f.user_id,
         AVG(CASE WHEN i.shown_at BETWEEN f.t0 - 14 AND f.t0 THEN i.was_clicked END) AS ctr_before,
         AVG(CASE WHEN i.shown_at > f.t0 AND i.shown_at <= f.t0 + 14 THEN i.was_clicked END) AS ctr_after,
         COUNT(CASE WHEN i.shown_at > f.t0 AND i.shown_at <= f.t0 + 14 THEN 1 END) AS impressions_after
  FROM first_click f
  JOIN ad_impression i
    ON i.user_id = f.user_id
   AND i.creative_id <> f.creative_id
  GROUP BY 1, 2
)
SELECT creative_id,
       COUNT(*) AS clickers,
       AVG(CASE WHEN impressions_after = 0 THEN 1.0 ELSE 0.0 END) AS share_no_return,
       AVG(COALESCE(ctr_after, 0) - ctr_before) AS trust_delta
FROM windows
WHERE ctr_before IS NOT NULL
GROUP BY 1
HAVING COUNT(*) >= 500
ORDER BY trust_delta ASC
LIMIT 50;

Two details in there are choices, not facts, so say them out loud. Coalescing a missing after-window to zero treats vanishing from the surface as the strongest possible non-click. That is defensible here, but it is not the same event as being shown ads and ignoring them. Hence share_no_return alongside it, so a reader can see how much of a creative's trust_delta comes from people leaving versus people staying and clicking less. And the after-window is half-open, strictly after t0, so the click itself does not land in both windows.

Rank creatives on their own click rate against this trust delta. The ones you want are high on the first and sharply negative on the second: they harvest clicks and leave the surface worse. Once you have that ranked list, treat the extreme tail as labels. Then train a supervised model on properties available before serving: headline text features, landing-page load time, the gap between headline entities and landing-page entities, and advertiser history. That converts a detection problem into a pre-serve blocking problem.

Two more caveats. Clickers are not a random sample, so the before-window is essential and matched non-clickers are a useful control. And reversing the sort surfaces creatives that increase subsequent trust, which is a better product idea than blocking.

checklist

Before you trust a proxy metric

• Mechanism check: can the treatment hurt the long-run outcome without moving this proxy

• Historical fit: has it been regressed against at least fifteen experiments with real long-run readouts

• Residual spread: is the prediction interval narrow enough to change a decision, not just to describe one

• Direction stability: does it point the same way for growth changes and monetization changes

• Gaming resistance: would a team optimising this proxy directly produce the outcome you want

• Refresh cadence: when was the mapping last refit, and has the product changed since

---

Holdbacks: the only honest long-run measurement

A proxy is an estimate. A holdback is a measurement. Long-run holdbacks are the mechanism by which a company keeps the ability to know what its own launches did.

Three kinds, and they answer different questions

tradeoff matrix

Choosing a holdback design

Choosing a holdback design

Design	Strength	Weakness	Use when
Long-run single-change holdback	Clean causal read on one launch, runs 6 to 12 months	Expensive per launch, only affordable for a few changes a year	The change is large, one-way, or contested
Global holdback	Measures the cumulative effect of everything shipped this year, catches slow drift no single test can see	Cannot attribute the loss to any one change	You want an annual answer to "did our year of shipping help"
Reverse holdback	Ships to everyone, then removes the feature from a small group later	Measures removal, not addition; adaptation confounds it	The feature is already live and you were never given a real test

A long-term holdback can measure the combined effect of a specified set of product changes. For example, randomly assign a planned fraction of eligible users to a stable baseline experience and follow both groups over the year. Document which changes the comparison covers. Both groups should receive required security, privacy, accessibility, and critical reliability fixes; a holdback is not a commitment to preserve known defects.

Its selling point sounds like an accounting impossibility, so be ready to explain it: twelve launches can each read plus 1 percent while the year in aggregate reads negative. Three mechanisms combine to produce that, and naming them is the argument for the instrument.

• Cannibalisation. Two features compete for the same session. Each is measured against a control that does not have either, so each banks the same shifted minutes, and the sum double-counts a gain that only happened once. That explains a shortfall, not a negative year; the other two are what deliver one.

• Cost outside the window. Each test is read on a window sized for its own fast effect, two to four weeks. The slow cost each launch seeds lands after that readout closes, so it falls outside every individual test while still landing inside the year.

• A shared resource nobody charges for. All twelve launches draw down attention, notification tolerance, and trust together, but each test can only see the marginal draw its own change made against a base the other eleven had not yet depleted. The pool empties even though no single withdrawal looks large.

A persistent holdback is one way to measure cumulative exposure to multiple changes over a common window. Its result depends on stable assignment, the actual treatment contrast, and interference between users. Long individual experiments and other suitable designs can also answer long-term questions; the holdback is useful when the combined package is the estimand.

Sizing it

Holdbacks are asymmetric: a small held-out group against a very large treated group. The small arm dominates the variance, and the sizing below makes that concrete.

Two constants first. The two-sided critical value at a 5 percent significance level is 1.959964, and the quantile giving 80 percent power is 0.841621. Raise the second to 1.2816 if you want 90 percent power instead. Second, the variance. The quantity you are testing is a difference of two proportions, whose variance is p1*(1-p1)/n1 + p2*(1-p2)/n2. Write the big arm as n2 = k * n1 and that becomes [p1*(1-p1) + p2*(1-p2)/k] / n1, which is the bracket the sizing formula evaluates. The large arm contributes only 1/k as much variance, so at k = 99 it is contributing about one percent of the total. The small arm is the whole cost.

At Nightjar's 4.2 million monthly actives and a 70 percent twelve-week retention baseline, a one percent holdback resolves about a 0.6 point retention difference. Carry the scaling law with you: sample size goes as one over the square of the effect, so halving the effect quadruples the holdback.

That scaling law is what turns "we would run a holdback" into a sentence with content:

"A one percent holdback on our base resolves roughly 0.6 points of retention. Halving the effect quadruples the holdback. So if the effect we care about is 0.3 points, we need about four and a half percent held out, or we should not claim we measured it."

Flag the approximation before someone else does. Fixing the treated-to-held-out ratio at 99 to 1 pins the allocation at a one percent holdback no matter what each row actually needs, so no row is exactly self-consistent. Below a one percent holdback the fixed ratio errs high; above it, low. On the first four rows that costs about one percent or less in relative terms, which does not matter. On the last row it costs about six percent, which does. At 0.25 points the real split is nearer 1 to 14 than 1 to 99, and the answer is 6.7 percent of the base, not 6.3. Solving for the fraction and its own ratio together gives 0.044, 0.175, 0.394, 1.595, and 6.735 percent. Either way 0.3 points lands near four and a half, 4.40 with the ratio fixed and 4.57 when you let it float.

Operating rules that people get wrong

• Keep assignment stable for the cumulative-effect question. Refreshing membership changes exposure histories and therefore changes the estimand. If reassignment is necessary, document it and design an analysis for the resulting histories.

• Define and enforce the treatment contrast. Track every surface covered by the holdback, along with shared fixes and other exceptions. Unplanned exposure reduces the difference between groups and can make the estimate difficult to interpret.

• Budget the cost explicitly. Under a proportional approximation, holding out one percent from a change worth twelve percent more revenue for treated users forgoes about 0.12 percent of baseline revenue. Weigh that opportunity cost against the value of the information and the operational cost of maintaining the comparison.

• Decide the readout date in advance and write it in the launch doc, because the pressure to peek and conclude early is enormous once the launch is already celebrated.

• Watch for contamination in social or shared-account products, where held-out users experience the change through people they interact with.

Interview tip: If asked how you would prove a launch from last year actually helped, answer "we would not be able to unless we kept a holdback, so I would start one now for this year's launches" instead of proposing an observational reconstruction that cannot work.

---

How to argue for a short-term loss

The analysis is the easy half. Getting a team to accept a quarter of worse numbers is the half that decides whether you have influence.

The weak script

"I think we should not ship the ad-load increase. The revenue gain looks real but I am worried about long-term effects on retention, and heavier ad loads have historically been bad for user experience. I would recommend we be cautious here."

Nothing in that is wrong and none of it is decidable. There is no number, no threshold, and no plan, so the meeting resolves in favour of whoever has the revenue target.

The stronger script

"Over thirteen weeks this banked an extra 0.17 USD of contribution per assigned user and raised weekly churn from 3.0 to 5.0 percent. Fifty-two weeks past the test, which is about fifteen months from when we started it, those net to about minus 0.01 USD per user. So this is a wash at best on the horizon we fixed before the test ran. It breaks even at 1.9 points of extra weekly churn and we measured 2.0. The interval on that comparison straddles zero, so the data do not resolve whether the effect clears that threshold. My recommendation is to not ship at four spots. Test a lower-exposure alternative, such as showing the fourth spot only to sessions longer than twelve minutes. The hypothesis is that this retains some of the additional revenue with less harm to retention. I would size both of those before proposing it as final. And put one percent into a twelve-month holdback so that next year we can actually settle this."

That version wins for four reasons. It uses one currency, and it gives a break-even the other side can argue with. It offers an alternative rather than a veto, and it proposes a way to settle the disagreement with evidence later.

Note what the script does not do: it quotes no revenue-capture percentage and no churn-cost fraction for the shaped alternative, because nothing in this analysis estimates either. Sizing them would take the session-length distribution, the share of impressions above the threshold, and an assumption linking exposure to hazard. And the panel carries a session count, not session length. Say "my expectation is" and name the work. An interviewer who asks where a confident 60 percent came from and finds you cannot derive it has learned something worse than that you did not know.

Pre-commit, in writing, before the launch

The most valuable habit here has nothing to do with statistics. Before the experiment starts, write down: the horizon, the decision rule at each outcome, the guardrails and their thresholds, and the holdback plan. Circulate it. Get the PM to agree.

This is not process hygiene. Once a result arrives the horizon becomes negotiable, and whoever wants to ship will find a horizon on which shipping is correct. This lesson's own table is the proof: the same evidence supports "ship, confidently" and "do not ship" depending only on which row you quote. Pre-registering the horizon converts a political argument into an arithmetic one.

concept flow

The short-term versus long-term decision sequence

1. 1
   Write the identity

   express the target metric as a product or sum, and name which term the change touches directly

2. 2
   Name the borrowed resource

   what shrinks, and does it regenerate if you stop

3. 3
   Choose the horizon and the margin definition

   before the readout, in writing, matched to the company's payback period

4. 4
   Read the test per assigned user

   revenue per assigned user and a survival curve, never per active user

5. 5
   Estimate the tail hazard

   from the last few weeks of the curve, and state it as an assumption

6. 6
   Compute banked plus forward value

   discounted contribution over the pre-registered horizon

7. 7
   Invert to a break-even

   the one parameter value at which the decision flips

8. 8
   Propose a shaped alternative

   capture most of the gain with less of the cost, rather than vetoing

9. 9
   Start the holdback

   so the next version of this argument has evidence instead of opinions

---

Common traps

Reporting per active user instead of per assigned user. If the treatment causes churn, dividing by survivors hides exactly the effect you are hunting. It can even make a harmful change look beneficial, because the users it drove away were the low-value ones. Always divide by the assignment denominator. The same trap hides inside SQL: a WHERE clause that drops rows with a NULL post-period metric is a survivors-only filter wearing a disguise.

Treating "retention was not significant" as "retention was unaffected." Two different failures hide behind that sentence and candidates collapse them into one. First, two weeks into a test nobody has a twelve-week retention outcome yet. That endpoint is censored, not underpowered, so no sample size fixes it and there is no minimum detectable effect to quote. Second, on the two-week retention you can observe, 20,000 users per arm resolves about 0.50 points against this dataset's 96.9 percent two-week baseline. Against a 70 percent twelve-week baseline it resolves about 1.3 points. The true week-two gap in the Nightjar panel is 0.17 points. Genuinely below the detectable effect, so invisible by construction, even though the same mechanism reaches 9.9 points by week thirteen. What you must not say is that the test was too small: 40,000 users is roughly fifteen times what a 5 point effect would need. Ask for the follow-up window and then the minimum detectable effect, never just "was it significant."

Extrapolating the observed hazard forever. In the Nightjar data the incremental hazard grows every week. The treated arm's weekly hazard is 0.030 plus 0.0018 times the week index. By the end of the quarter it sits more than two points above control, and it is still climbing. Projecting that growth to infinity produces a catastrophic and unbelievable number. Projecting a flat hazard at the week-13 level, as this lesson does, is the defensible middle. It is also the optimistic side of the middle, since the real hazard was still rising when the test ended. Say which assumption you made and show the answer under one alternative.

Comparing revenue on one side to margin on the other. The short-term gain is usually quoted in revenue, because that is what the dashboard shows. The long-term loss lands in contribution. Convert both, or the comparison is meaningless.

Choosing the horizon after seeing the result. The single most common way this analysis gets corrupted, and almost never deliberate. Pre-register it.

Reading a forward horizon as a total horizon. A close cousin of the same mistake, and easier to make by accident. When your banked term already covers the test weeks, a "52 week" forward projection is fifteen months of elapsed time, not twelve. Label the axis with total elapsed weeks before anyone quotes a verdict off it.

Using a proxy that the treatment can bypass. Optimising ad load against "revenue per session" cannot fail: revenue per session is what the change increases. A proxy has to be able to say no.

Refreshing the holdback population. Re-randomizing each quarter destroys the accumulated difference the holdback exists to measure. Membership is permanent for the life of the study.

Assuming every short-term win is a trap. Real free wins exist: faster load times, fixing a broken funnel step, a better search index. If you cannot articulate the mechanism by which a change costs something later, do not invent one. Manufactured skepticism reads as inexperience, not rigour.

Ignoring the competitive clock. A change that destroys value over 24 months can still be correct if the company will not exist in 24 months without the revenue. Name the runway constraint rather than pretending it away.

---

Quick self-check