Describe an end-to-end ML project

# Example, end-to-end answer: Personalized Home Feed Ranking for a Marketplace Below is a structured, first‑person example that hits each dimension. Numbers are illustrative; tailor them to your experience. ## 1) Business objective - Problem: The home feed showed popular items with simple heuristics. It over-indexed on clicks and missed purchases, hurting GMV and seller exposure fairness. I led a project to build a two-stage retrieval + ranking system to personalize the feed for buyers. - Objective: Increase GMV and purchase conversion without violating latency/cost budgets or deprioritizing new/long-tail sellers. - Constraints: p95 latency ≤ 150 ms end-to-end; infra cost increase ≤ 20%; maintain category diversity and a minimum exposure to new sellers. Small numeric framing: A 2% GMV lift on a $5M/day baseline ≈ $100k/day, enough to justify added infra costs if guardrails hold. ## 2) Stakeholders and roles - Product (Discovery PM): Prioritization, success criteria, launch plan. - Data/ML: Me (lead), 1 data scientist for measurement, 1 MLE for serving. - Data engineering: Event pipelines, feature store, catalog joins. - Infra/SRE: Kubernetes resources, autoscaling, observability, incident response. - Analytics/Experimentation: Test design, power analysis, guardrails. - Legal/Privacy: Retention windows, user consent, data minimization. - Seller ops/support: Fairness concerns, change management. ## 3) Success metrics and guardrails - Primary KPI: GMV per session and purchase conversion (orders/session). - Secondary: Add-to-cart rate, average order value, buyer retention D7. - Quality/fairness: Category diversity, new-seller exposure share, buyer complaint rate. - Operational guardrails: p95 latency ≤ 150 ms; error rate ≤ 0.1%; infra cost ≤ +20%. - Attribution window: Purchases within 7 days of impression (also report 24h for quicker readouts). Optimization target used in ranking: Expected GMV per impression E[GMV] = P(purchase|user,item) × price × margin. ## 4) Data and pipelines - Sources: - Event logs: Impressions with position, clicks, add-to-cart, purchases (joined via impression_id), dwell time. - Catalog: Item price, category, brand, availability, shipping time, seller rating. - User profile: Cohort, recency/frequency, preferred categories, device. - Real-time signals: Recent views/carts (24h), trending items, inventory. - Labels: - Positive: A purchase within 7 days of impression; secondary label for click within session. - Negatives: Exposed but not purchased. To handle class imbalance, downsample negatives at ~1:10 with weights. - Bias mitigation: - Position bias addressed in training/eval via inverse propensity weights (IPS) from randomized slots we reserved (~1–2% traffic) and historic randomized experiments. - Pipelines: - Batch (daily): ETL in Spark; feature engineering; offline store (warehouse) + online store (low-latency KV). - Stream: Kafka for real-time features (recent activity counts), computed with Flink and pushed to the online feature store. - Orchestration & quality: Airflow DAGs with freshness SLAs; data contracts, null/volume/anomaly checks; feature store ensures training-serving schema parity. ## 5) Modeling choices - Baseline: Heuristic blend of popularity × recency × price filters. - Architecture: Two-stage system. 1) Retrieval (candidate generation): Two-tower embeddings trained on click/purchase co-occurrence (BPR loss). ANN index (Faiss/ScaNN) returns ~500 candidates per user in <10 ms. 2) Ranking: Gradient-boosted trees (LightGBM/LambdaMART) optimizing for purchase/NDCG, with a final calibration step for probability (isotonic). We rank by expected GMV. - Features (examples): - User: category affinity scores, spend band, device, geo. - Item: price, discount depth, shipping SLAs, seller quality, novelty. - User×Item: category match, price vs user spend band, recency of user–seller interactions. - Context: time-of-day, day-of-week, seasonality, inventory. - Cold start: - New users: popularity + content-based similarity; collect signal via lightweight exploration ε ≈ 5%. - New items/sellers: content-based features + boosted exposure quota during warm-up. - Why this stack: - Two-tower retrieval scales and supports real-time personalization. - GBDTs for ranking gave strong performance, fast iteration, interpretability, and low serving latency compared to deeper models. ## 6) Training setup - Splits: Time-based; train on last 60 days, validate on next 7, test on subsequent 7. - Losses: - Retrieval: BPR/softmax on implicit feedback; hard negative mining from recent impressions. - Ranking: LambdaMART for NDCG@K; also trained a logistic variant for purchase probability used to compute E[GMV]. - Hyperparameters: Optuna for search; early stopping based on NDCG@50. - Regularization: Tree depth constraints, min child weight, L2; feature bagging. - Imbalance: Negative downsampling with inverse sampling weights. - Calibration: Isotonic regression on a held-out set to improve probability–to–GMV alignment. - Frequency: Daily retraining; embeddings weekly, with hot-fixes as needed. - Compute: Distributed training on CPU cluster for GBDT; GPU for two-tower embeddings. - Leakage controls: No post-impression signals in features; label windows strictly after impression timestamp. ## 7) Evaluation methodology - Offline metrics: - Retrieval: Recall@500; coverage across categories/sellers. - Ranking: NDCG@20, log loss, AUC; expected GMV per 1,000 impressions; IPS-weighted variants to counter position bias. - NDCG formula: DCG@K = Σ_{i=1..K} (rel_i / log2(i+1)); NDCG@K = DCG@K / IDCG@K. - Offline→online correlation: - Track metric correlations over previous experiments; choose NDCG@20 (IPS-weighted) and expected GMV as best predictors of online GMV lift. - Experimentation: - A/A to validate parity and variance; then 50/50 A/B, 2–4 weeks. - Guardrails: latency, error rate, complaints, category diversity, new-seller exposure, returns rate. - Stats: Clustered SE at user level; CUPED for variance reduction; pre-registered stop rules to avoid peeking. - Small numeric example: If baseline CVR = 6.0% and target relative lift = 3%, absolute delta = 0.18 pp. With observed session variance, we estimated needing ~3–5M sessions/variant for 80% power (illustrative; compute from your data). ## 8) Infra and serving - Architecture: - Online feature store (KV/Redis) for low-latency joins; offline warehouse for training. - Retrieval service hosts ANN index; ranking service (gRPC) loads a Treelite-compiled GBDT model. - End-to-end budget: retrieval ~10 ms, features ~40 ms, ranking ~20 ms, network ~30 ms, p95 < 150 ms. - Deployment: - Model registry (MLflow); CI/CD with canary rollout (5%→25%→50%→100%); automatic rollback on SLO breach. - Monitoring: - Real-time: CTR/CVR, GMV/session, latency/error, feature freshness. - Data quality: training-serving skew checks, drift (PSI/KL) alerts, missing-value spikes. - Post-release: guardrail dashboards and anomaly detection. ## 9) Trade-offs, failures, and debugging - Click vs purchase conflict: Early model optimized CTR and hurt conversion (clickbait items). Fix: optimize for expected GMV and add dwell/quality features; calibrate probabilities. - Position bias: Offline gains didn’t translate online. Fix: IPS-weighted training/eval; allocate small randomized exposure to keep propensities fresh. - Training-serving skew: A real-time feature was computed differently online, causing mismatch. Fix: unify feature definitions in feature store, add parity tests in CI. - Latency spikes: Large feature sets increased p95 latency. Fix: feature ablation + caching; trimmed 15% features with minimal lift impact. - Fairness: Long-tail seller exposure dropped. Fix: post-rank re-ranking with diversity/fairness constraints and minimum exposure quotas; track fairness KPIs. - Inventory mismatch: OOS items occasionally ranked. Fix: real-time availability feed + hard filter before ranking. - Debugging toolkit: SHAP for feature contribution sanity; slice analysis by user/seller segments; join coverage auditing; replay tests on recorded traffic. ## 10) Impact - Online A/B (illustrative): - +3.5% GMV/session, +2.8% purchase conversion, +0.4% AOV; guardrails met (p95 latency 132 ms, error rate 0.06%, cost +12%). - New-seller exposure maintained within ±0.5 pp; category diversity slightly improved. - Rollout: 100% after 3 weeks; incident-free. ## 11) What I’d do differently to improve impact - Invest earlier in unbiased data collection (more randomized slots) to tighten offline→online correlation and speed iteration. - Build a unified retrieval+ranking online learning loop (contextual bandits) to balance exploitation and exploration, especially for cold-start sellers. - Move to periodic embedding refresh (daily) and streaming re-ranking for high-velocity events. - Introduce multi-objective optimization explicitly (GMV, diversity, fairness) with transparent knobs for product to tune. - Expand explainability and self-serve dashboards for stakeholders; faster root-cause analysis and safer experimentation. How to adapt this to your story - Swap in your domain (search, ads, fraud, supply/demand forecasting). - Keep the structure; bring 2–3 quantified results; highlight 1–2 real failures and your fix. - Tie decisions to constraints (latency, cost, privacy, fairness) and show end-to-end ownership.