# Design a Deduplicated Personalized News Feed Design a news aggregation service that ingests articles from many publishers and serves personalized feeds. Every publisher gives each article a unique URL, but several publishers may report the same underlying event. The feed should retain useful source diversity while avoiding repeated stories about the same event. ### Constraints & Assumptions - Article ingestion and feed reads are continuous and horizontally scaled. - URL equality detects exact duplicates from one source but not cross-publisher duplicates. - Deduplication may be approximate and should expose a confidence score. - Personalization uses a candidate set plus ranking; model details are secondary to system boundaries. - A breaking story may evolve, so clusters can merge or split over time. ### Clarifying Questions to Ask - Does deduplication hide all but one article or group alternatives into one story card? - What freshness window defines the same event? - Which mistakes are worse: merging different events or repeating one event? - What latency is acceptable for ingestion and feed retrieval? ### Part 1 - Ingestion and story identity Design article normalization, exact deduplication, feature extraction, candidate matching, and story-cluster assignment. #### What This Part Should Cover - Stable article and publisher identities - Canonical URL handling without relying on URL alone - Text, entity, time, and media similarity signals - Versioned clusters and confidence-aware decisions ### Part 2 - Personalized serving Explain how story clusters become feed candidates and how the serving path chooses a representative article without discarding source alternatives. #### What This Part Should Cover - Story-level ranking rather than article-level repetition - Representative selection and source diversity - Cached candidate retrieval with freshness - Pagination stable under cluster updates ### Part 3 - Quality, scale, and correction Discuss approximate-neighbor search, hot breaking stories, human or automated corrections, evaluation, and failure recovery. #### What This Part Should Cover - Precision and recall of clustering - Feedback for false merges and false splits - Idempotent ingestion and replay - Monitoring of duplicate exposure and feed quality ```hint Separate article identity from story identity Keep each publisher's article as an immutable record, then assign it to a versioned story cluster used by the feed. ``` ### What a Strong Answer Covers - A credible cross-publisher entity-resolution pipeline - Personalized serving at the story level with transparent source choices - Explicit false-merge versus false-split trade-offs - Scalable indexes, correction paths, and measurable quality ### Follow-up Questions 1. How would you prevent a single fast publisher from defining the cluster narrative forever? 2. What happens to pagination when two story clusters merge? 3. Which online metric best reveals that users are seeing the same event repeatedly?