Walk through a key project

# How to structure your answer (works well for ML Engineer technical screens) Use a crisp STAR/CAR structure plus metrics and trade-offs: - Situation/Context: One sentence on the product and why the problem matters. - Task: Your objective, target metrics, and constraints. - Actions: Your technical approach and leadership/ownership. - Results: Quantified impact, time to value, adoption. - Reflection: Trade-offs, challenges, and what you’d change now. Pro tip: Anchor with 2–3 top metrics and 2–3 design decisions. Avoid "we did everything"—be specific about your part. --- # Example answer you can adapt: Personalized Feed Ranking Revamp 1) Problem and context - A mobile news app needed to improve home feed engagement without increasing latency or clickbait. Baselines showed flat CTR and declining session length as the catalog grew. - Goal: Lift CTR and dwell time while keeping p50 latency under 60 ms and p99 under 150 ms. Maintain quality/health via diversity and bounce-rate guardrails. 2) My role - Lead ML Engineer (IC): owned problem framing, modeling, offline/online evaluation design, and the online serving changes. Partnered with a data engineer (pipelines/feature store), backend engineer (service integration), and a PM/Content lead. 3) Technical approach - System design: 2-stage recommender (candidate generation → ranking) with a lightweight re-ranker for diversity. - Candidate generation: Two-tower deep retrieval using user and item embeddings trained with sampled softmax on implicit feedback (clicks, dwell > 10s). Approximate nearest neighbor (ANN) index for sub-10 ms retrieval. - Ranker: Gradient-boosted trees (XGBoost) for speed/interpretability, then migrated to a DNN once stable. Features included recency, topic embeddings, publisher reliability, personalization signals, device/network, and session context. - Re-ranker: Determinantal point process (DPP)-inspired heuristic to promote topical diversity and down-rank clickbait via a calibrated quality score. - Data and labeling - Events: Impressions, clicks, dwell time, hides, follows. Bot filtering applied. - Labels: Positive if dwell ≥ 12s or saved; negatives sampled with time decay to counter exposure bias. - Split: Strict time-based split to prevent leakage; user-level grouping to avoid cross-user contamination. - Offline evaluation - Metrics: AUC, NDCG@10, calibration error (ECE), and coverage. Diversity measured via topic entropy. - Example: NDCG@10 improved from 0.42 → 0.47; ECE dropped from 0.09 → 0.04. - Online evaluation - A/B test with 50/50 holdout, two-week duration, min detectable effect 2% CTR with power 0.8. - Primary: CTR, Avg dwell per session. Guardrails: p50/p99 latency, bounce rate, diversity, publisher coverage. - Deployment/infra - Feature store with TTL; all online features mirrored offline to reduce train–serve skew. - ANN via Faiss HNSW; per-user request budget <15 ms for retrieval, <45 ms for ranker. - Canary rollout with auto-revert, real-time metric watchdogs. 4) Success metrics and targets - Targets: +5% CTR, +3% dwell per session, p50 <60 ms, p99 <150 ms, no increase in bounce rate. - Achieved: CTR +8.7% (p<0.01), dwell +5.1%, p50 45 ms, p99 128 ms, bounce rate -1.3%, topic diversity +6%. 5) Outcome - Rolled out to 100% of traffic within 3 weeks. Sustained gains for 90 days. Incremental revenue +6% (ads tied to sessions). Infra cost +2% due to ANN indexing offset by caching and batch-precomputed features. - Documented playbook for future launches; upstream teams adopted the feature store patterns. 6) Major trade-offs - Accuracy vs latency: Chose XGBoost for ranker initially; DNN only after caching and vectorization stabilized, keeping p99 in check. - Personalization vs diversity: Added re-ranking constraints; a pure CTR objective overfit to clickbait. We included a quality score and topic-coverage penalties. - Exploration vs exploitation: Introduced epsilon-greedy exploration in top-k (ε≈0.05) to learn about tail content without tanking CTR. 7) Unexpected challenges and fixes - Data leakage: Real-time features (e.g., recent clicks) were computed differently online vs offline. Fixed by unifying transformations in the feature store and adding train–serve skew checks. - Non-stationarity (news is fast): Concept drift caused weekly degradation. Added recency-weighted training, daily incremental retrains, and time-decayed features. - Feedback bias: Popular items got more exposure. Mitigated with inverse propensity weighting (IPW) in offline eval and diverse exploration online. - Latency spikes at p99: ANN probes bursty under high concurrency. Tuned HNSW efSearch and added per-user cache warmup. 8) What I’d do differently now - Multi-objective optimization: Train with a composite objective (CTR, dwell, diversity, quality) or use constrained optimization to meet guardrails by design. - Counterfactual/off-policy evaluation: Use IPS/DR estimators to better predict online impact and reduce A/B cost. - IPS: E[ w_i * y_i ] where w_i = 1 / p_logging(a_i|x_i). Helps debias offline estimates for exploration traffic. - Causal metrics: Penalize clickbait via dwell-normalized CTR or long-term retention uplift (e.g., 7-day sessions per user). - Better cold-start: Pretrain item embeddings from text/images with contrastive learning to reduce new-article ramp time. --- # Tips to tailor your own example - Swap the domain (e.g., ads CTR, content moderation, fraud detection) but keep the structure. - Always include: the 1–2 key architectural choices, 2–3 metrics with numbers, 1–2 trade-offs, and a clear lesson learned. - Validate with guardrails: canary rollout, kill switch, holdback cohorts, latency SLOs, and a revert plan.