Netflix Personalized Recommendations: Choosing a Success Metric and Guardrails

Netflix suggests titles to each member based on what that member has watched and how they have rated it. You are the product manager who owns this recommendation system. How would you measure whether it is succeeding? Explain what the system is for, the metric you would hold it to, the supporting and guardrail metrics around that metric, and how you would tell a real improvement from noise. ```hint Watch what happens after the click A recommendation that earns a click is not necessarily a good one. Think about which member behavior, after a recommended title starts, separates a suggestion that paid off from one that only looked appealing. ``` ### Clarifying Questions - Does "the recommendation system" mean every personalized surface (which home-screen rows appear, the order of titles within them, suggestions shown when a title ends) or one surface such as the home page? - Is this a health check of the live system, or the criteria for deciding whether a new recommendation model should replace the current one? - Which business outcome matters most to leadership right now: keeping subscribers, adding new ones, or growing viewing? - Should the metrics work across all markets and devices, or for one market or device type first? ### What a Strong Answer Covers - A clear purpose for the system, for members and for Netflix's subscription business, and why that makes retention the outcome it ultimately serves. - A precisely defined north-star metric that captures satisfying viewing started from recommendations, rather than raw clicks or raw hours. - Supporting metrics along the member's path from browsing to finishing a title, plus guardrails such as failed browsing sessions, early abandonment and catalog diversity. - A way to attribute changes to the recommendations rather than to new releases or seasonality: controlled experiments, long-running holdouts and segment cuts. - Trade-offs between short-term engagement and long-term satisfaction, and how members with little or no viewing history are handled. ### Follow-up Questions - A new ranking model raises plays started from the home page, but the share of those plays that members finish stays flat. Would you ship it? - How would you judge recommendation quality for a member who joined yesterday and has no history? - Effects on cancellations take months to show up. Which faster metric would you rely on, and how would you check that it actually predicts retention?