Design a Topic-Based News Aggregator with Fast Removal

Design a news aggregator that periodically pulls articles from the APIs of tens of thousands of publishers. A supplied black-box classifier assigns one or more topics to each article. Users follow multiple topics, potentially hundreds, and receive a combined feed. Explain pagination, deduplication, read performance, and how to remove an article identified as false from every user's feed quickly. Assume recent article IDs are already cached for each topic; explain how you could improve read speed beyond that baseline. ### Constraints & Assumptions - Publisher collection is periodic API polling. - The topic classifier is supplied; designing or training it is out of scope. - Articles may belong to several topics, and a user may follow hundreds of topics. - Feed order, freshness targets, and the exact meaning of duplicate content must be clarified. - Rapid removal must address cached and precomputed feeds, not only the primary article record. ### Clarifying Questions to Ask - Is the feed chronological, ranked, or a combination, and what consistency should pagination provide? - Does deduplication mean identical publisher IDs, identical URLs, or substantially identical stories across publishers? - How fresh must newly collected articles be, and how quickly must a removal take effect? - What should happen to an in-progress paginated session when follow preferences change? ### Part 1 — Collect, Classify, and Build Feeds Describe ingestion, article identity, topic indexes, and feed generation for a user following many topics. Explain a read optimization beyond caching each topic's recent IDs. #### What This Part Should Cover - Bounded publisher polling, retries, and ingestion deduplication. - Canonical article identity and multiple topic memberships. - Read-time merging versus materialized user or cohort feeds. - The cost of freshness, follower fanout, and hot topics. ### Part 2 — Paginate, Deduplicate, and Remove Articles Specify cursor behavior and duplicate handling across topic lists and pages. Then trace an urgent article removal through storage, caches, and feed reads. #### What This Part Should Cover - A stable ordering key and an explicit pagination consistency model. - Deduplication by identity without losing distinct stories through unsupported assumptions. - Authoritative visibility status and invalidation of derived results. - A removal guarantee that accounts for stale materializations and read caches. ```hint Deleting an ID from every user's precomputed feed may be slow. Consider what the final serving path must check even while background cleanup continues. ``` ### What a Strong Answer Covers The design should quantify the work of combining hundreds of topic lists and explain what data is precomputed to reduce that work. It should make pagination stable under new ingestion and ensure removed articles are filtered from the serving path even before every derived feed has been rebuilt. ### Follow-up Questions - How would you prevent one high-volume publisher from dominating ingestion capacity? - What if an article appears in several followed topics at different positions? - How would you handle an article restored after an erroneous removal?