Design a News Aggregator That Builds Personalized Feeds from Publisher APIs

Design a news aggregator. The system pulls articles from many publishers through each publisher's API (major news outlets and similar sources) and gives every user a feed of relevant articles, similar to a general-purpose news aggregation app. The system does not host article content: it stores only metadata such as title, summary, category, and the URL of the original article, and users read the full article on the publisher's site. ### Constraints and Clarifications - Traffic numbers are not given. Estimate the ingestion volume and read traffic yourself, and state what each number leads you to build or deliberately not build. - Publisher APIs are third-party systems: they can be slow, unavailable, or rate limited, and they differ in how they page through new articles. ### Clarifying Questions - How does a user express relevance: following publishers and categories, or implicit personalization from reading behavior? - How fresh must a feed be after a publisher releases an article? - Can one article belong to several categories? - How far back should a user be able to scroll in the feed? - Do publisher APIs provide a cursor, a timestamp filter, or only page numbers for fetching new articles? ### Part 1 — Ingesting Articles From Publisher APIs Design how articles are fetched and stored. Explain how often each publisher is polled, how one polling run knows when to stop, and how the system behaves when a publisher's API fails or throttles requests. ```hint Remember where you stopped Think about what per-publisher state lets each polling run fetch only what is new, and what makes it safe to repeat a run after a crash. ``` #### What This Part Should Cover - A scheduler and worker design with per-publisher progress tracking, and a justified polling frequency. - Stop conditions for a polling run and idempotent storage so re-fetched articles are not duplicated. - Handling of rate limits, timeouts, and outages, with monitoring that detects a silent publisher. ### Part 2 — Building and Serving Feeds Design how a user's feed is assembled and paginated from the stored metadata. ```hint Precompute per topic, merge per user Consider building ordered lists per publisher and per category, and what the read path must do when a user follows several of them. ``` #### What This Part Should Cover - The data model for articles and categories, and how new articles propagate to feed structures. - The read path: merging the lists a user follows, removing duplicates when an article appears in more than one list, and stable pagination. - Bounding storage for lists that grow forever, and the capacity estimate that justifies the chosen level of optimization. ### What a Strong Answer Covers - Clear functional requirements, with metadata-only storage respected throughout. - Estimates that are computed and then used to make explicit decisions, including decisions not to optimize. - A reliable ingestion pipeline with per-publisher progress, idempotency, backoff, and isolation between publishers. - A feed design with deterministic ordering, correct deduplication, stable cursors, and bounded storage. - Proactive coverage of failure handling and observability, not only when asked. ### Follow-up Questions 1. A publisher changes an article's title after publication. How does the correction reach users' feeds? 2. How would you add personalized ranking on top of the chronological merge without rewriting the pipeline? 3. How would you detect that one publisher has silently stopped returning new articles? 4. How does your design change if a publisher offers push notifications (webhooks) instead of an API you poll?