Design a News Aggregator That Clusters Articles into Ranked Stories

Design a news aggregator: a service that collects articles from many news publishers, recognizes when different publishers cover the same event, ranks the resulting stories, and shows users a feed of current news. The report gives only this one-line prompt, so settle the scope with the interviewer before designing. ```hint Same event, many articles Many publishers cover the same event within minutes of each other, often with near-identical wire copy. Decide how the system recognizes that articles belong together and what the feed shows for that group. ``` ```hint Shared work versus per-user work Separate the work that is the same for every user (ingesting, grouping, scoring) from the work that differs per user, and decide where and how often each runs, given how many more reads than writes there are. ``` ### Constraints and Clarifications - Not given: the number of sources, articles per day and users, the freshness target, whether the feed is personalized, and which regions and languages are served. Ask, or state your assumptions. ### Clarifying Questions - Where do articles come from: publisher feeds (RSS or Atom), publisher APIs or pushes, or crawling article pages? Is full text stored, or only headline, summary and link? - Is the feed the same for everyone in a region, personalized per user, or both? - How quickly must a newly published article appear: within minutes, or is an hourly refresh acceptable? - How many sources, articles per day and daily active users? Which regions and languages? - Are topic pages, search or breaking-news notifications in scope? ### What a Strong Answer Covers - Settled scope: sources, personalization, freshness target and scale, with estimates - An ingestion pipeline that fetches from many sources politely, isolates slow or broken ones, and normalizes articles - Duplicate detection and clustering of articles into stories, with the precision and recall trade-off - A ranking approach combining recency, coverage, source quality and engagement, plus personalization, with a clear split between precomputed and request-time work - A data model and read path that serve feeds with low latency under heavy read load (caching, stable pagination) - Edits and retractions, failure handling, scaling and monitoring ### Follow-up Questions - A major event breaks and thousands of articles about it arrive within minutes. What happens at each stage, and how quickly does the story reach the top of the feed? - A publisher edits a headline or retracts an article after it has been clustered and shown. How does the change propagate? - How do you evaluate whether a ranking change makes the feed better? - How would you add breaking-news push notifications without notifying a user twice about the same story?