Design a Personalized Recommendation System: Retrieval, Ranking and Evaluation

Design a recommendation system. This is an ML system design round, so the focus is the machine learning: how the problem is framed, which data and labels train the models, which models retrieve and rank the items, how the system is evaluated, and how it is served and monitored. The report names only the topic, so what is recommended, on which surface and at what scale are open: settle them with the interviewer first. If no product is specified, practice with a personalized home feed that shows each user items from a large and growing catalog. ```hint Count the items Consider how many items exist, how many a heavy model could score within one request, and what that implies for the shape of the system. ``` ```hint Name the label Decide which logged user action the model should predict, and ask whether a click alone shows that the user valued what they were shown. ``` ### Clarifying Questions - What is being recommended, on which surface (home feed, "more like this", notifications), and how many slots does the surface show? - How many users and items are there, how fast do new items appear, and what latency budget does a request have? - What is the business goal: time spent, retention, purchases, or a balance that also protects creators and diversity? - Which interactions are logged: impressions, clicks, time spent, purchases, likes, hides, reports? - Are there safety or age-appropriateness rules on what may be shown to whom? ### What a Strong Answer Covers - Framing: the objective, the labels built from logged behavior, and the online metrics they should move - A multi-stage architecture: candidate generation from several sources, ranking, and re-ranking with policy rules - User, item, context and interaction features, built without leakage or training-serving skew - Model choices for retrieval and ranking, with their trade-offs - Offline evaluation on a time-based split and online A/B testing with guardrail metrics - Cold start, position bias, feedback loops and diversity, serving latency, and monitoring with retraining ### Follow-up Questions - A newly published item has no interactions yet. How does it get shown at all, and how do you learn whether it is good? - The offline metric of a new ranker improved, but the A/B test shows no gain. What could cause that, and how do you investigate? - The feed keeps showing the same few popular items. How do you add diversity and exploration without hurting engagement? - How do you correct for position bias in the logged clicks used for training?