Lead an ML project under ambiguity

# How to structure a top-tier answer Use STAR plus L (Situation, Task, Actions, Results, Learnings): - Situation and Task: 2–3 sentences to set scope, scale, and stakes. - Actions: Walk through decisions from problem framing to deployment and monitoring, calling out trade-offs and stakeholder alignment. - Results: Quantify business impact, latency, reliability, and cost. - Learnings: Include a failure and what you changed. Highlight reproducibility, fairness, and ethics under time pressure. Tip: Tie every ML choice to a business KPI or risk, and quantify with concrete numbers. # Answer blueprint you can adapt - Situation: Product, users, baseline metric and pain point. - Goal and KPIs: Primary business KPI, proxy metrics, explicit guardrails and SLOs. - Stakeholders: Who, cadence, decision log, approvals (privacy or legal). - Data and privacy: Sources, joins, quality, retention, minimization, regionalization. - Modeling: Baselines, candidate models, trade-offs (latency, interpretability, cost), final choice and why. - Offline eval: Metrics, validation splits, leakage checks, calibration, slicing. - Experiment: Hypothesis, unit of randomization, power or MDE, guardrails, pre-registered plan. - Deployment: Feature store, model registry, rollout or canary, SLOs, rollback. - Monitoring: Business, model, data, fairness and drift; alert thresholds. - PM pushback: The conflict, data you used to influence, decision outcome. - Impact: Lifts, latency, cost, reliability, adoption. - Failure and learnings: Root cause, fixes, process and tooling for reproducibility and ethics. # Worked example: Notification ranking and send-time optimization Situation and task - Product had heuristic push notifications. Baseline CTR 7.5 percent, weekly unsubscribe 1.1 percent, WAU growth flat. I led an ML system to decide which notifications to send and when, with strict latency and privacy constraints. Goal and success metrics - Primary KPI: WAU uplift target 1.5 percent. - Online primary: relative CTR lift 5 percent or more. - Guardrails: unsubscribe rate decrease 10 percent or more, complaint rate not worse, session length not worse. - Platform SLOs: p95 inference latency under 60 ms, error rate under 0.1 percent, cost per 1k predictions under 3 cents. Stakeholder alignment - Weekly triad with PM and Eng; biweekly privacy and legal check-ins. One-pager set scope, KPIs, guardrails, timelines, and go or no-go criteria. Logged decisions in a shared design doc. Data sourcing and privacy constraints - Sources: server logs for sends, opens, unsubs; message metadata; device and locale. No raw message content in training, only coarse categories to minimize sensitive exposure. - Privacy: user IDs hashed; 90-day TTL for user-level features; regional data residency; differential privacy style aggregation for population counters where feasible; DPIA completed and approval from privacy counsel before any ramp. - Data quality: contracts for event schemas; unit tests and anomaly alerts for missing or shifted features. Model selection and rationale - Framing: two-stage decision. Stage 1 eligibility model to predict whether sending is net-positive for a user now. Stage 2 ranker to order eligible notifications for a user. - Baselines: recency or frequency heuristic and logistic regression. - Candidates: logistic regression, gradient boosted trees, and a lightweight two-tower for content and user embeddings. - Choice: gradient boosted trees with monotonic constraints on frequency features for interpretability and controls, plus isotonic calibration. It matched deep model lift within 0.5 percent but was 3 times faster and cheaper. Offline evaluation - Time-based splits; label defined as open within 24 hours; leakage checks on future-aware features. - Metrics: AUC 0.78 to 0.86; NDCG at 5 up 14 percent; Brier score down 9 percent; calibration error 2.7 percent to 1.8 percent. - Slices: consistent gains across platform, locale, and activity segments. - Simulated guardrail: survival model estimated unsub would drop with frequency capping informed by the eligibility model. Online experiment design - Unit: user-level randomization. 50 or 50 split, starting at 10 percent canary. - Power: for baseline CTR 8 percent and minimum detectable effect 0.4 percentage points (5 percent relative), approximate sample size per group is about 150 thousand exposures for 95 percent confidence and 80 percent power. We planned a 2-week run given traffic. - Pre-registered analysis plan: primary metric CTR; guardrails unsubscribe and complaint rate; CUPED to reduce variance; cluster-robust errors; one interim check with alpha spending to avoid p-hacking. Deployment and reliability - Features in an online feature store with 10-minute freshness SLAs; offline or online parity tests. - Inference service on CPU with ONNX export, 8-bit quantization, and caching of top features per user to reduce latency. - Canary at 1 percent, then 10 percent, 50 percent, 100 percent with auto-rollback on guardrail breaches. Post-launch monitoring - Dashboards: CTR, WAU, unsubscribe, complaints; model metrics (calibration, lift vs. baseline shadow), data drift (PSI on key features), and fairness slices by locale and platform. - Alerts on p95 latency, error rate, and unsubscribe spikes; kill switch to revert to heuristic policy. Strong PM pushback and how I influenced the decision - PM wanted to optimize purely for CTR and increase daily send limits, arguing for faster growth. I showed a simple LTV model: incremental CTR gain from more sends was outweighed by projected lifetime loss from increased unsubscribes. I proposed a multi-objective target that penalized sends when modeled unsubscribe risk exceeded a threshold and set a hard cap per user per day. We ran a 10 percent holdout comparing pure CTR optimization versus the penalized objective. The penalized objective delivered 5.9 percent CTR lift with 13.2 percent lower unsub (versus 7.1 percent CTR with 3.8 percent higher unsub), and higher predicted LTV. We adopted the penalized objective with frequency caps. Quantified impact - Online at 50 percent ramp: CTR plus 6.4 percent relative (95 percent CI about 4.1 to 8.7). Weekly unsub minus 12.7 percent (CI about minus 9.3 to minus 16.1). WAU plus 2.1 percent. - Reliability: p95 latency from 85 ms to 32 ms via quantization and caching. Error rate 0.06 percent. - Cost: cost per 1k predictions from 5.8 cents to 2.3 cents. At about 80 million predictions per day, saved roughly 2.8 thousand dollars per day, about 1.0 million dollars per year. Failure and what I changed - In the first 10 percent canary, Spanish-language locales saw an 8 percent rise in unsubscribes. Root cause: send-time features trained on global data favored early morning times that conflicted with local quiet hours and cultural patterns. Fixes: added locale-aware quiet hours and country-specific time-of-day features; segmented calibration by locale; added a pre-launch checklist requiring guardrail simulation by region. The re-run removed the unsub spike and maintained CTR gains. Reproducibility, fairness, and ethics under time pressure - Reproducibility: end-to-end pipeline in version control; data versioning with checksums; model registry with lineage and immutable artifacts; seed control and deterministic training; time-based splits encoded as config. Every experiment was tagged and reproducible via a single command. - Fairness: no use of sensitive attributes; monitored equality of opportunity proxy by platform and locale; if any slice showed guardrail breach, auto-throttled sends for that slice and required a review before removal. - Ethics and privacy: data minimization and retention limits; regional residency; no raw message content in training; documented model card with intended use, limitations, and known failure modes; DPIA completed before scale-up. When a leadership deadline compressed timelines, we scoped to the safest subset of features, required passing guardrails and privacy sign-off before any ramp, and deferred riskier features to a later release. # Experiment design guardrails and quick math - Sample size for two-proportion CTR test (approximate): n per group is about 2 times (z alpha-half plus z beta) squared times (p times (1 minus p) plus p' times (1 minus p')) divided by (p' minus p) squared. With p equals 0.080 and p' equals 0.084, z alpha-half about 1.96 and z beta about 0.84, n per group is about 150 thousand exposures. - Guardrails: predefine stop or go criteria, block sequential peeking without alpha spending, and include fail-safe: unsubscribe plus complaints cannot worsen beyond thresholds. Use user-level randomization and cluster-robust errors to avoid inflated significance. # Common pitfalls and how to avoid them - Offline or online mismatch: use time-based splits, calibration, and counterfactual or replay simulation where feasible. - Data leakage: ban post-event features and watch time windows. - Novelty effects: run at least 2 weeks or across full cycles. - Latency blowups: choose simpler models first, quantify cost per 1k predictions, and use quantization or distillation. - Fairness regressions: always slice and alert; document in a model card and set enforcement thresholds. This structure demonstrates end-to-end ownership, quantifies trade-offs, and shows principled decision-making under real-world constraints.