Success Metrics and Measurement Plan for Netflix's AI Investments

How would you measure whether Netflix's use of AI is a success? ```hint Split before you measure "AI at Netflix" is not one feature. Decide which applications you are judging and whom each one serves before you name a single metric, then ask how you would know the AI caused any change you see. ``` ### Clarifying Questions - Which AI do you mean: member-facing personalization such as recommendations and artwork, a new generative feature such as conversational search, or internal uses such as streaming optimization and content operations? - Are we judging AI that is already running, or deciding whether to keep investing in it? - Should the answer weigh business results such as retention and revenue, member experience, internal efficiency, or all three? ### What a Strong Answer Covers - A breakdown of AI at Netflix into specific applications, each tied to the member or business outcome it should move. - A layered metric set linking model quality, member behavior and business results, with a clear north-star metric and guardrails. - Causal measurement such as A/B tests and long-term holdouts rather than before-and-after comparisons, and how to handle slow outcomes like retention. - AI-specific guardrails and costs: diversity of what members watch, trust, errors in generated content, latency and compute spend. ### Follow-up Questions - A new recommendation model lifts viewing hours in a two-week test, but a long-term holdout shows no change in cancellations. What do you conclude? - How would you measure an AI tool used by internal teams, where a member A/B test is impossible? - Hours viewed go up while member surveys say it has become harder to find something worth watching. Which signal do you trust?