Launch Decision Metrics for Testing a New Spotify Track Recommender

Your team at Spotify has built a new algorithm for recommending songs. As the product manager, which metrics would you use to make the call on launching it? ```hint Look past the first session A recommender can win plays today and lose listeners next month. Think about what it should change in listening over several weeks, and what it could quietly make worse. ``` ### Clarifying Questions - Which surface does the algorithm power: a personalized playlist, autoplay after an album ends, radio, or recommendations on the home screen? - What is it meant to improve: total listening, discovery of new artists, or retention? - Will it be tested against the current algorithm in a randomized experiment, and how long can that test run? - Are both Free and Premium listeners in scope? ### What a Strong Answer Covers - A stated goal and one primary decision metric that reflects listener value rather than raw plays - Supporting metrics on how listeners respond to recommended tracks, plus longer-term listening and retention - Guardrails covering total listening, subscriptions and ad revenue, negative feedback, artist diversity and system performance - An experiment design with a randomization unit, a duration long enough for novelty to fade, and a ship rule agreed in advance - How to decide when results conflict across metrics or listener segments ### Follow-up Questions - Plays from recommendations rise but total listening per user is flat. Do you ship? - How would you tell whether the model is finding new music for people or replaying songs they already know? - New listeners improve and long-time Premium subscribers get slightly worse. What do you do?