Describe leading an ambiguous ML project end-to-end

# What is actually being scored This is a leadership question wearing an ML costume. The interviewer is checking five things: 1. **Can you create clarity?** A vague ask ("stop the bad accounts", "make notifications better") becomes a written problem statement, a label definition, a primary metric, and guardrails. 2. **Do you own the whole system?** Data and privacy, modeling, evaluation, rollout, monitoring, and the on-call runbook, not just the notebook. 3. **Do you decide with numbers?** Every trade-off is tied to a business KPI or a risk, with concrete thresholds. 4. **Can you disagree productively?** You changed a senior stakeholder's mind with evidence, and you also committed to a decision you lost. 5. **Are you honest?** One real failure, its root cause, and the process change that followed. Answer in STAR-L: Situation, Task, Actions, Results, **Learnings**. Keep Situation and Task to two or three sentences, spend most of the time in Actions, and always land the quantified Result. --- # Worked example A: real-time signup risk scoring under a vague mandate This example is written to cover parts 1, 3, 5, 6, 7, 8, and 11. Example B below covers parts 2, 4, 9, and 10 in more depth. In a real interview you tell **one** story that hits all eleven. ## Situation A surge in fake and bot signups was driving spam, support load, and downstream abuse. Leadership asked us to "block bad accounts at signup" before a large marketing launch. The ask was genuinely ambiguous: no definition of "bad", no friction budget, no agreed success metric, no latency or cost constraint. ## Task Convert the ask into a precise, measurable ML problem and lead delivery end-to-end across modeling, infrastructure, and policy. ## Actions ### 1. Scoping: from a vague ask to a written contract **Problem statement.** Predict the probability that a new account will be disabled for a policy violation within 7 days, and use that score to route the signup to one of three actions: allow, challenge (SMS / 2FA), or block. Writing the label definition down was the single highest-leverage act of the project. "Disabled within 7 days for a policy reason" is checkable, backfillable, and it forced the trust and safety team to agree on what counts. It gave a 1.8% positive rate across 50M historical signups. **Success metrics, agreed in writing with PM, engineering, legal, and finance:** | Dimension | Target | | --- | --- | | Business | Abusive accounts at D1 and D7 down 50% or more | | Friction guardrail | False blocks of legitimate users at or below 0.3%; signup completion down no more than 1.0 pp | | Model quality | ROC-AUC at or above 0.92; PR-AUC at least 2x the rules baseline | | Calibration | Brier score beating the base-rate constant predictor (see the calibration note below) | | Latency | p95 under 20 ms, p99 under 40 ms at 1k QPS; 99.9% availability | | Cost | Inference at or below $2,000 per month; feature store reads at or below $0.15 per 1,000 predictions | | Fairness | Ratio of false-positive rate across the top 5 regions at or below 1.5x; no sensitive attributes as features | ### 2. Data and labels - Time-based splits to prevent leakage: 10 months train, 1 month validation, final month as a blind test. Random splits would have leaked future abuse-ring behaviour backwards. - Features: device and network signals (IP /24, ASN, proxy and Tor flags), velocity counters (signups per device and per IP per hour and per day), user-agent entropy, hashed and aggregated email-domain reputation, time of day and week, geolocation consistency. - Privacy: no raw IP retained beyond 30 days, network features hashed and aggregated, PII encrypted at rest, retention policy documented and approved before training started. ### 3. Model selection and the trade-offs | Model | ROC-AUC | PR-AUC | p95 latency | | --- | --- | --- | --- | | Logistic regression | 0.86 | 0.24 | 2 ms | | Random forest | 0.90 | 0.33 | 18 ms | | **LightGBM (chosen)** | **0.94** | **0.49** | **12 ms** (p99 24 ms) | Because positives were rare, PR-AUC was the primary offline metric and ROC-AUC was reported only as a secondary. Class imbalance was handled with class weights, benchmarked against focal loss. **Interpretability trade-off.** I enforced monotonic constraints on the risk-coded features (more recent failed verifications must never decrease risk) and shipped per-decision SHAP values. Monotonicity cost nothing measurable in PR-AUC and bought two things worth more than a fractional point: appeals agents could explain a block to a user, and legal accepted the model faster. **Calibration note, and a correction worth internalising.** With a 1.8% base rate, a constant predictor that always outputs 0.018 already scores a Brier of 0.018 x 0.982 = **0.0177**. Any Brier figure in the 0.09 to 0.13 range would therefore be far *worse* than predicting the base rate, so quoting "Brier improved from 0.128 to 0.093" on a rare-event problem shows the metric was never sanity-checked. Isotonic regression took us from 0.0165 to **0.0121** against that 0.0177 reference. Always quote a rare-event Brier next to its base-rate reference, or use a Brier skill score instead. ### 4. The decision policy and how the thresholds were set Score s in [0, 1]: - **Allow** when s < 0.20 - **Challenge** when 0.20 <= s < 0.70 - **Block** when s >= 0.70 Thresholds came from an expected-cost calculation with a cost matrix estimated jointly with finance and PM: a missed abuser costs about $2.10 in downstream remediation, a wrongly blocked legitimate user costs about $7.50 (lost user plus a support contact), and a challenge costs about $0.06 with a 93% completion rate among legitimate users. At the chosen cutoffs on the blind test: - The **block** tier catches 32% of eventual abusers at 86% precision, which is a false-block rate of **0.10% of legitimate signups**, comfortably inside the 0.3% guardrail. - The **block plus challenge** tiers together intercept **70%** of eventual abusers, at the cost of challenging 3.1% of legitimate users. - Expected cost per signup fell 41% against the rules baseline. Note how the two tiers are reported separately. A common mistake is to quote a single "recall 0.72 at FPR 0.18%" for a three-way policy: at a 1.8% base rate that pair implies roughly 88% precision at 72% recall, which is flatly inconsistent with a PR-AUC of 0.49, and it hides the fact that a challenge and a block impose very different costs on a user. Report the operating point of each action separately and check that precision, recall, and PR-AUC can coexist. **Cost and infrastructure trade-off.** LightGBM served in-process with a warm feature cache, against an online feature store with 10 ms p95 reads. Projected inference cost of about $1.7k per month at peak QPS, inside the $2k budget. ### 5. Stakeholders, checkpoints, and disagree-and-commit - **PM:** agreed the business OKR and the friction budget (no more than 0.3% false blocks, no more than a 1.0 pp drop in signup completion). - **Engineering:** agreed the SLOs and, importantly, the degradation mode. If inference fails, fall back to allow-plus-challenge-only rather than fail closed and block real users. - **Legal and privacy:** no raw PII in features, hashed and aggregated network features, 30-day retention, DPIA documented, and a fairness guardrail of a regional FPR ratio at or below 1.5x. - **Decision gates:** PRD and risk doc sign-off, then model card and fairness report, then a shadow-launch review, then the canary go / no-go, then the post-experiment readout. **Disagree-and-commit.** PM wanted to skip the shadow phase entirely and start blocking before the campaign. I recommended two weeks of shadow to measure the real false-block rate, and showed the expected-cost curve for being wrong about the block threshold. We settled on one week of shadow with blocking limited to the top 0.5% of scores. I still thought one week was too short and said so, then committed: I tightened the automatic rollback triggers and raised the block threshold for launch week so that the shorter shadow was survivable. That is the shape of the story to tell. Disagreement backed by numbers, a compromise, then genuine commitment with compensating controls rather than quiet sabotage. ### 6. De-risking and rollout - **Shadow, 7 days.** Read-only scoring in production. Score distribution was stable against training (PSI 0.08), and delayed labels gave a live estimate of the tier operating points before anyone was affected. - **Canary and ramp.** 10% canary, then 50%. Mid-risk users were challenged only; blocks were limited to the highest tier at first. - **Rollback criteria, automatic, falling back to allow-plus-challenge-only:** - False-block rate above 0.3% for 5 consecutive minutes, or above 0.25% for 30 minutes - Signup completion worse than -0.8 pp against control for 30 minutes - p99 latency above 50 ms for 10 minutes, or inference error rate above 0.5% for 5 minutes Writing rollback triggers *before* launch is what turns a risky launch into a reversible one, and it is the detail most candidates leave out. ### 7. Monitoring - **Model:** tier-level precision and recall on delayed labels, backfilled PR-AUC, expected calibration error. - **Data:** PSI on the score and on key features, alerting above 0.2, plus null-rate and entropy checks. - **Fairness:** false-positive rate and challenge rate by region, alerting if the max ratio exceeds 1.5x. - **System:** p50 / p95 / p99 latency, QPS, error rate, cache hit rate, cost per 1k predictions. - **Business:** abusive-account incidence at D1 and D7, manual review queue depth, user-reported spam. - On-call runbook with a one-click policy downgrade and a feature-flag kill switch. ## Results (90 days post-launch) - Abusive accounts down 58% at D1 and 54% at D7 against control. - Manual review hours down 42%; spam reports down 31%. - Signup completion down 0.2 pp, inside the 1.0 pp budget. - Roughly $3.6M annualized savings, validated with finance. - Blind-test ROC-AUC 0.94, PR-AUC 0.49; live calibration stable. - p95 14 ms, p99 28 ms at 1.1k QPS; 99.97% availability; about $1.8k per month. - Max regional FPR ratio 1.32x, inside the guardrail. Model card published. ## Reflection (part 11) - **Pre-commit the experiment design.** A decision matrix and a minimum detectable effect agreed before building would have saved a week of debate. - **Make the cost-weighted utility the primary metric from day one.** Threshold conversations were painful until finance and PM were looking at dollars rather than AUC. - **Ship schema validation and drift monitors before shadow**, not during it under time pressure. - **Start the model card and risk register at kickoff.** Doing so later became the gating item for legal review. - **Shadow across a seasonal boundary.** Mild drift appeared after a holiday event (PSI about 0.22) that a longer shadow would have caught. --- # Worked example B: notification ranking and send-time optimization Use this one if your strongest story is a growth or ranking system. It carries the sub-parts example A treats lightly: data and privacy (part 2), formal experiment design (part 4), a real failure (part 9), and reproducibility, fairness, and ethics under time pressure (part 10). **Situation.** Push notifications were rule-driven. Baseline CTR 7.5%, weekly unsubscribe 1.1%, WAU flat. I led an ML system deciding which notification to send and when. **Metrics.** Primary KPI: WAU up 1.5%. Online primary: relative CTR lift of 5% or more. Guardrails: unsubscribe rate down 10% or more, complaint rate not worse, session length not worse. SLOs: p95 inference under 60 ms, error rate under 0.1%, cost under $0.03 per 1k predictions. **Data sourcing and privacy (part 2).** Server logs for sends, opens, and unsubscribes; message metadata; device and locale. Deliberately **no raw message content** in training, only coarse categories, to minimise sensitive exposure. User IDs hashed, 90-day TTL on user-level features, regional data residency, aggregated population counters, DPIA completed and privacy counsel sign-off obtained before any ramp. Event schemas were pinned by data contracts with unit tests and anomaly alerts on missing or shifted features. **Modeling.** Two stages: an eligibility model for whether sending now is net positive for this user, then a ranker over eligible notifications. Candidates were logistic regression, gradient-boosted trees, and a lightweight two-tower model. GBTs won: they came within 0.5% of the deep model's lift while being roughly 3x cheaper and faster, and monotonic constraints on frequency features gave the controls the PM needed. Isotonic calibration on top. **Offline evaluation.** Time-based splits, label = open within 24 hours, explicit leakage checks on any future-aware feature. AUC 0.78 to 0.86, NDCG@5 up 14%, Brier down 9%, calibration error 2.7% to 1.8%, with gains holding across platform, locale, and activity slices. **Online experiment design (part 4).** User-level randomization (notifications spill across sessions, so session-level randomization would leak treatment). 50/50 split starting from a 10% canary. Pre-registered analysis plan: primary metric CTR, guardrails on unsubscribes and complaints, CUPED for variance reduction, cluster-robust standard errors, and a single interim look with alpha spending so that peeking does not inflate the false-positive rate. **Sizing the test, with the arithmetic done correctly.** For a two-proportion test with baseline p = 0.080 and a minimum detectable effect of 0.4 pp (p' = 0.084), at 95% confidence and 80% power: n per group = (z_{alpha/2} + z_beta)^2 * [p(1-p) + p'(1-p')] / (p' - p)^2 = (1.96 + 0.84)^2 * (0.0736 + 0.0769) / (0.004)^2 ~= 74,000 exposures per group (about 148,000 total) The pooled shortcut, `2 * (z_{alpha/2} + z_beta)^2 * p(1-p) / (p'-p)^2`, gives about 72,000 per group, which is the same answer. The factor of 2 in that shortcut stands in for the second group's variance, so applying the 2x **and** summing both variance terms double-counts and lands on about 148,000 per group. That is a common and expensive slip: it doubles your test duration for no statistical reason. Quote the per-group number and the total explicitly so nobody has to guess which one you mean. **Deployment.** Online feature store with a 10-minute freshness SLA and offline / online parity tests, ONNX export with 8-bit quantization on CPU, top-features cached per user. Canary at 1%, then 10%, 50%, 100%, with auto-rollback on any guardrail breach. **PM pushback (part 5).** PM wanted to optimise CTR alone and raise the daily send cap. I built a simple LTV model showing the incremental clicks from extra sends were outweighed by the lifetime value lost to higher unsubscribes, then proposed a penalised objective that discounts a send when modelled unsubscribe risk crosses a threshold, plus a hard per-user daily cap. We settled it with a 10% holdout rather than with opinions: pure CTR optimisation gave +7.1% CTR but +3.8% unsubscribes, while the penalised objective gave +5.9% CTR with 13.2% *fewer* unsubscribes and higher predicted LTV. We shipped the penalised objective. **Impact (part 8).** At the 50% ramp: CTR +6.4% relative (95% CI roughly 4.1% to 8.7%), weekly unsubscribes -12.7% (CI roughly -9.3% to -16.1%), WAU +2.1%. p95 latency 85 ms to 32 ms via quantization and caching, error rate 0.06%. Cost per 1k predictions $0.058 to $0.023, which at about 80M predictions per day is roughly $2,800 per day, about $1.0M per year. **Failure and what changed (part 9).** In the first 10% canary, Spanish-language locales showed an 8% *rise* in unsubscribes. Root cause: send-time features were trained on globally pooled data, so the model favoured early-morning slots that collided with local quiet hours and daily rhythms. Fixes: locale-aware quiet hours, country-specific time-of-day features, calibration segmented by locale, and a pre-launch checklist item requiring guardrail simulation per region. The re-run removed the spike and kept the CTR gains. The process change mattered more than the model change, and that is the point to land. **Reproducibility, fairness, and ethics under time pressure (part 10).** - *Reproducibility:* pipeline in version control, data versioned with checksums, a model registry with lineage and immutable artifacts, seed control and deterministic training, splits encoded as config. Every experiment reproducible from a single tagged command. - *Fairness:* no sensitive attributes as features, equality-of-opportunity proxies monitored by platform and locale, and automatic throttling of any slice that breached a guardrail pending human review. - *Ethics and privacy under pressure:* when leadership compressed the timeline, we did not cut the guardrails. We cut *scope*, shipping the safest feature subset, keeping privacy sign-off and guardrail passes as hard gates on any ramp, and deferring the riskier features to a later release. "We shrank the launch instead of the safeguards" is the sentence interviewers remember. --- # Common pitfalls in this answer - **No numbers.** "It improved engagement" scores near zero. Give AUC or PR-AUC, p95 and p99 latency, cost per month, guardrail values, and the confidence interval on the lift. - **Metric that does not fit the problem.** ROC-AUC as the headline on a 1.8%-positive problem, or a Brier score quoted without its base-rate reference. - **A model with no policy.** A score is not a decision. Say what action each score range triggers and what each error costs. - **Rollout with no rollback.** Name the trigger, the threshold, the duration, and the fallback state. - **Offline / online mismatch.** Time-based splits, calibration, and replay simulation. Ban post-event features. - **Novelty effects.** Run at least two weeks, or across a full weekly cycle. - **Conflict with no resolution.** The pushback story needs the data you brought, the decision that was made, and, if you lost, what you did about it anyway. - **A sanitised failure.** "I worked too hard" is not a failure. Give a real regression, its root cause, and the process change it produced.

Rubric for the interviewer and self-check for the candidate. Structure the answer as STAR-L (Situation, Task, Actions, Results, Learnings), with the Actions section walking the full lifecycle in order: framing to data and privacy to modeling to offline and online evaluation to stakeholder alignment to staged rollout to monitoring, then a quantified Result and an honest Learning. A strong answer shows all of the following, and each is a distinct scoring line: 1. A vague ask converted into a written label definition, a primary metric, and explicit guardrails. 2. Data provenance, label validity, leakage control via time-based splits, and a named privacy or retention constraint. 3. A model choice justified against at least two competing candidates on accuracy, latency, interpretability, and cost, with numbers. 4. Metrics appropriate to the base rate (PR-AUC over ROC-AUC for rare events; a calibration figure quoted against its base-rate reference) plus a real experiment design: unit of randomization, power or MDE arithmetic, guardrail metrics, and a pre-registered plan that controls for peeking. 5. Named stakeholders, decision gates, and one conflict resolved with evidence, including a disagree-and-commit moment where the candidate lost and committed anyway with compensating controls. 6. A staged rollout (shadow, canary, ramp) with rollback criteria that were written before launch. 7. Monitoring across four axes: model quality, data drift, fairness slices, and system health. 8. Impact quantified in business, latency, reliability, and cost terms. 9. A genuine failure with a root cause and the process change that followed. 10. Reproducibility and fairness practices that survived time pressure, ideally by cutting scope rather than safeguards. 11. Specific hindsight, not a platitude. Red flags: no numbers anywhere; a model with no decision policy attached; a launch with no rollback trigger; a metric that cannot coexist with the other metrics quoted (for example a precision, recall, and PR-AUC triple that is arithmetically impossible); a conflict story with no resolution; a sanitised failure.