Design an ML system that generates a personalized, ordered 10-track playlist from a candidate pool of 10,000 tracks in real time. The entire playlist is chosen before playback begins; the model cannot adapt midway through the session. Both track selection and ordering matter. Available signals include user demographics and listening history, track embeddings and metadata, request context such as time, device, and entry point, plus 28 days of playback, skip, replay, save, and implicit session-sequence logs. The product objective is user satisfaction, observed indirectly through playlist completion, especially early skips, saves, and follow-up listening. ### Constraints & Assumptions - Do not collapse the task into ten independent top-score choices without addressing sequence quality, repetition, and transition effects. - Training labels are behavioral proxies and may be affected by exposure, position, autoplay, and the existing recommender. - The serving design must meet a clarified real-time latency budget and degrade safely if sequence reranking is unavailable. - Treat demographic features as sensitive inputs requiring a justified benefit, governance, and fairness review. ### Clarifying Questions to Ask - How are the 10,000 candidates produced, and may the playlist repeat an artist or a recently played track? - Is satisfaction optimized per request, across the full session, or for longer-term retention? - Which events are reliable negative feedback, and how are accidental plays or offline playback handled? - What exploration traffic or randomized logging is available for counterfactual evaluation? ```hint Separate item relevance from slate construction A fast pointwise model can narrow the pool, while a sequence-aware reranker reasons about the tracks already placed in the playlist. ``` ### What a Strong Answer Covers - Objective and label construction, delayed outcomes, position and exposure bias, and multi-objective trade-offs. - Candidate filtering, item scoring, sequence-aware slate generation, diversity constraints, and deterministic fallback behavior. - Training examples from sessions, negative sampling, leakage prevention, offline evaluation, and online experimentation. - Feature freshness, embedding and model serving, latency controls, monitoring, feedback loops, fairness, and retraining. ### Follow-up Questions - How would you distinguish a bad track from a good track placed at a bad position? - How can the model trade off immediate completion against discovery and future listening? - What would you log so that a later model can evaluate alternative playlists without replaying the user session?