## Design a Recommendation System End-to-End You are asked to design a large-scale recommendation system that powers a personalized feed — for example, a short-video "For You" feed or an e-commerce "recommended for you" surface. The system must serve a personalized ranking to hundreds of millions of users, handle cold start for brand-new users and items, adapt to user actions in near real time, and optimize for long-term engagement while controlling for harmful feedback loops. Walk through the full design: candidate generation and ranking, the real-time and batch feature pipelines, training/serving/evaluation, and the monitoring and guardrails that keep the system healthy and safe. ### Constraints & Assumptions - **Scale:** hundreds of millions of monthly active users; an item corpus from the millions (e-commerce SKUs) to billions (UGC short videos). Tens of thousands of feed requests per second at peak. - **Latency:** end-to-end feed-serving budget on the order of a few hundred milliseconds (p99), split across retrieval, ranking, and re-ranking. - **Freshness:** the system should reflect a user's most recent actions within seconds, and surface newly created items within minutes to hours. - **Objective:** the north-star is long-term engagement / retention, not just immediate clicks. Assume you can log rich feedback (impressions, clicks, watch time, likes, follows, purchases, skips, hides, reports). - Assume a feature store, a stream-processing layer, an offline data warehouse, and an experimentation (A/B) platform are available to build on. ### Clarifying Questions to Ask Before designing, a strong candidate scopes the problem: - What is the primary success metric and over what horizon — same-session watch time, next-day retention, 28-day retention, or revenue? How do we trade short-term clicks against long-term value? - What is the content type and corpus size (UGC video vs. a finite catalog of products)? This drives retrieval strategy and cold-start severity. - What are the hard guardrails — safety/policy violations, legal constraints, advertiser brand-safety, creator fairness — and are any of them non-negotiable filters versus soft penalties? - What is the latency budget and the cost ceiling (GPU/serving spend per thousand requests)? - What signals can we log and how delayed are the labels (a click is instant; a "good session" or a return-visit label is delayed by hours or days)? - What is the diversity/exploration appetite — how much are we allowed to show non-greedy recommendations to learn about new users and items? ### Part 1 — Candidate Generation & Ranking Architecture Design the multi-stage funnel that turns a corpus of millions-to-billions of items into a ranked feed for a single request. Cover how candidates are retrieved, how they are scored, and how the final list is assembled. ```hint Funnel shape Recommenders are almost always a **funnel**: a cheap, high-recall retrieval stage narrows billions of items to ~hundreds/thousands, then a progressively more expensive **ranking** (and light **pre-ranking**) stage scores that shortlist, and a final **re-ranking** stage applies diversity and business rules. Reason about the latency/quality budget at each stage. ``` ```hint Retrieval techniques For retrieval, think **two-tower** (separate user and item encoders → embedding dot-product → ANN index such as HNSW/IVF) for personalization, plus complementary sources: item-item co-visitation / graph walks, "follow"/social, trending, and an exploration source. You usually blend several retrieval channels. ``` ```hint Ranking objective The ranker should predict **expected utility**, not a single proxy. Multi-task heads (CTR, watch-time/dwell, like, follow, skip, report) combined into one score let you balance objectives and counter degenerate optimization of any single proxy. Sequence/transformer models over the user's recent-action history are state of the art here. ``` #### What This Part Should Cover - The multi-stage funnel (retrieval → pre-rank → rank → re-rank) and why each stage exists, with rough candidate counts and latency per stage. - Concrete retrieval methods (two-tower + ANN, co-visitation/graph, trending/exploration) and how they are blended. - The ranking model: inputs (user/item/context/cross features), architecture choices, and a multi-objective formulation rather than a single click proxy. - Re-ranking concerns: diversity, de-duplication, business/policy constraints, and how the final ordering is produced. ### Part 2 — Feature Pipelines (Real-Time and Batch) Design the feature pipelines that feed both training and serving. Address how batch and streaming features are computed, stored, and kept consistent between offline training and online inference. ```hint Two pipelines, one store Distinguish **batch features** (long-term user preferences, item popularity, embeddings recomputed daily) from **real-time features** (last-N interactions, current session, item velocity over the last minutes). A **feature store** with online + offline views is the standard way to serve both from one definition. ``` ```hint The leakage trap The single most important correctness property is **point-in-time / time-travel correctness**: training labels must only see feature values that were available *before* the label event, or you leak the future. Pair this with **train/serve parity** so the same feature logic runs in both paths (ideally compute-once, log features at serving time). ``` #### Clarifying Questions for this Part - How fresh must real-time features be — seconds (session adaptation) or sub-second (abuse/velocity signals)? - Do we log features at serving time (the gold standard for parity) or recompute them offline for training, and what are the cost trade-offs? #### What This Part Should Cover - Clear separation of batch vs. streaming feature computation, with example features in each. - A feature store providing online/offline views and the mechanism for train/serve parity. - Point-in-time correctness / leakage avoidance as an explicit design property. - Freshness, backfill, and handling of missing/late features at serving time. ### Part 3 — Cold Start, Training, Serving & Evaluation Cover the model lifecycle: how the system handles new users and new items (cold start), how the models are trained and refreshed, how they are served at low latency, and how the system is evaluated both offline and online. ```hint Cold start = content + exploration New **users** lean on context (geo, device, time, referrer), popularity, and fast onboarding signals; you adapt within the session. New **items** lean on **content-based embeddings** (text/image/video encoders) so they can be retrieved before behavioral data exists, plus a controlled exploration budget (e.g., a bandit) to gather early feedback without flooding the feed. ``` ```hint Offline ≠ online Offline metrics (AUC/logloss for calibration, NDCG/recall@k for ranking) are necessary but not sufficient. The decision metric is an **online A/B test** on the long-term objective, watched alongside **guardrail metrics**. Be explicit that offline wins frequently fail to replicate online. ``` #### What This Part Should Cover - Cold-start strategy for both new users and new items, including content-based features and exploration (bandits / exploration budget). - Training cadence (batch retrain vs. incremental/online updates), data sampling (negative sampling, debiasing exposure), and label construction including delayed labels. - Serving architecture: low-latency ANN retrieval + ranking service, caching, embedding refresh, model rollout, and graceful fallbacks. - Evaluation: offline metrics (ranking + calibration), online A/B testing on the long-term metric, and the role of guardrail metrics in ship decisions. ### Part 4 — Monitoring, Guardrails & Feedback Loops Design the monitoring and safety layer. Address operational health (drift, skew, latency), content/ecosystem safety (spam, harmful content, fairness), and the feedback loops created by recommending based on what the model itself surfaced. ```hint Feedback loops A recommender trains on data it generated, so it can amplify its own biases — popularity rich-get-richer, filter bubbles, and degenerate engagement loops. Counter with **diversity constraints**, **exploration**, **negative feedback** (skips/hides/reports as strong signals), and **exposure debiasing** (e.g., inverse-propensity weighting) in training. ``` ```hint What to monitor Separate **system health** (latency p95/p99, error rates, feature freshness, training-serving skew, data drift) from **ecosystem/safety health** (harmful-content rate, complaint/block rate, creator and topic concentration, fairness across user groups). Long-term-engagement optimization needs explicit guardrails or it will exploit short-term proxies. ``` #### What This Part Should Cover - Identification of harmful feedback loops (popularity bias, filter bubbles, engagement traps) and concrete mitigations (diversity, exploration, debiasing, negative-feedback signals). - Operational monitoring: feature/label drift, training-serving skew, latency and availability SLOs, and alerting. - Safety and fairness guardrails: spam/abuse demotion, policy filtering, fairness across user and creator groups, and ecosystem-health metrics. - How guardrails interact with the optimization objective (hard filters vs. soft penalties; guardrail metrics gating launches). ### What a Strong Answer Covers Across all parts, strong answers connect the components into one coherent system rather than describing them in isolation: - A clear **funnel architecture** with sensible latency/quality budgets at each stage and justified retrieval/ranking choices. - **Multi-objective optimization** tied to a stated long-term north-star, with explicit awareness that immediate-click proxies diverge from long-term value. - **Data correctness** as a first-class concern: train/serve parity, point-in-time correctness, and exposure/position debiasing. - Honest treatment of the **offline-vs-online gap**, with A/B testing and guardrail metrics driving launch decisions. - A credible **safety and feedback-loop** story, not just an accuracy story — diversity, exploration, fairness, and ecosystem health. - Reasonable **scale/cost reasoning** (ANN at billions of items, caching, model/embedding refresh cadence) consistent with the stated constraints. ### Follow-up Questions - Immediate clicks and long-term retention conflict — clickbait wins CTR but hurts retention. How would you design the label, objective, and evaluation to optimize for long-term value despite delayed and sparse retention signals? - The two-tower retrieval model is trained on items the system already showed. How do you correct for this **exposure/selection bias** so retrieval doesn't collapse onto a popular subset? - A new creator or seller complains their fresh, high-quality items never get traffic. How would you quantify and address cold-start fairness without flooding users with untested content? - Your A/B test shows a watch-time win but a rise in user reports and content concentration. How do you decide whether to ship, and how would you encode that trade-off so the decision is repeatable?