Choosing Success Metrics for a Personalized Recommendation Row on Netflix
Company: Netflix
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
Netflix is introducing "Top Picks," a new feature that uses machine learning to recommend titles to each member. What metrics would you track to decide whether it is working?
```hint Follow the member past the click
A recommendation can earn plays that members quickly abandon, or plays they would have found in another row anyway. Think about what a good result looks like after the click, and how you would tell new viewing apart from viewing that simply moved.
```
### Clarifying Questions
- Where does Top Picks appear: as a new row on the home screen, in place of an existing row, or as its own section, and on which devices?
- What should it achieve: help members choose something faster, surface titles they would not have found, or keep members subscribed longer?
- Can we run an A/B test against the current home screen, or will it ship to everyone at once?
- Is it purely personalized, or will the business also use it to promote new originals?
### What a Strong Answer Covers
- A goal tied to how Netflix makes money, with one primary metric that reflects members finding something they enjoy
- A funnel from impression to play to satisfied viewing, keeping engagement with the row separate from outcomes for the member
- Guardrails for cannibalization of other rows, early abandonment, negative feedback and app performance, plus recommendation health such as diversity and catalog coverage
- How an A/B test would read these metrics, including the unit of randomization and how slow-moving retention is handled
- Trade-offs between short-term clicks and long-term satisfaction
### Follow-up Questions
- Plays from Top Picks climb sharply, but total viewing per member does not move. What happened, and is the feature a success?
- How would you detect over several months whether Top Picks narrows what members watch, and what would you do about it?
- Retention takes months to show a change. Which short-term metric would you use in its place, and how would you check that it actually predicts retention?
Overview: An analytics question about evaluating Top Picks, a new machine learning recommendation feature on Netflix. It tests choosing a goal and a primary metric, separating clicks on the row from satisfied viewing and retention, setting guardrails for cannibalization and quality, and reading the results in an A/B test.