Design a News Aggregator That Ingests Publisher Articles Into User Feeds

Design a news aggregator: a service that continuously collects articles from many news publishers and shows users a feed of current news that they can browse, for example by topic. The report gives only the name of the problem, so the kinds of sources, the degree of personalization, the scale and the freshness target are open: settle them with the interviewer before designing. ```hint Thousands of sources, one schedule Think about how often each source really needs to be checked, and how to avoid downloading and processing what has not changed. ``` ```hint One story, many outlets Ask what the feed should look like when many publishers report the same event within an hour. ``` ### Clarifying Questions - How are articles collected: publisher RSS or Atom feeds, partner APIs, or crawling web pages? How many sources are there? - How soon after publication must an article appear in feeds? - Is the feed the same for everyone in a region or topic, or personalized from follows and reading history? - Do we store and show full article text, or a headline, summary and image with a link to the publisher? - How many users and feed requests per second at peak, and is search in scope? ### What a Strong Answer Covers - Functional and non-functional requirements and scale estimates agreed before the design - An ingestion pipeline: per-source scheduling, conditional fetches, parsing and normalization, rate limits and source failures - Duplicate detection and grouping of the same story across publishers - Feed construction and ranking: topic feeds, personalization, and precomputation versus assembly at read time - Storage, caching and stable pagination for a read-heavy workload with breaking-news spikes - Failure handling and monitoring of freshness and source health ### Follow-up Questions - A major breaking story multiplies the read rate within minutes. What breaks first, and how do you protect it? - A source starts publishing duplicates or spam. How do you detect and contain it? - How do you paginate a feed that keeps changing as new articles arrive, without duplicates or gaps? - How would you add full-text search over articles?