Design a News Aggregator: Data Structures, Database Choice and Failure Handling
Company: Rippling
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Design a news aggregator: a service that continuously collects articles from many news publishers and shows users a feed of recent news, with each item linking to the publisher's article. The report gives the problem's name and the flow of the round: clarify the traffic (QPS) first, draw the main data structures, then go deep on them; the interviewer then asked which database you would choose and how each service handles failures. Scale, freshness and personalization requirements are open, so settle them before designing.
```hint Where articles come from
Think about how new articles enter the system when publishers update at very different rates, and what stops the same article from being stored twice.
```
```hint Shape the data for the read
Ask which query the feed page runs most often, and which stored structure answers it without scanning all recent articles.
```
### Clarifying Questions
- How many publishers, how many new articles per day, how many daily users, and what feed read QPS at peak?
- Is the feed the same for everyone in a category or region, or built per user from followed topics and publishers?
- How soon after a publisher posts must an article appear?
- Do we store full text, or only title, summary, image and a link out to the publisher? How long are articles kept?
- Is the order purely by recency, or ranked? Should several publishers' stories about the same event be grouped?
- Is search over articles in scope?
### What a Strong Answer Covers
- Requirements, and QPS and storage estimates that drive the later choices
- Ingestion: polling publishers' feeds at per-source rates, normalization, and deduplication by URL and by content
- The core data structures (article record, per-topic time-ordered indexes, user follows) and how a feed page is assembled from them, with pull versus push decided from the numbers
- A database choice for each kind of data, justified by its access pattern and growth
- Failure handling for every service: scheduler, fetchers, queue, processing, stores, cache and feed service
- Caching, pagination of a changing feed, and monitoring of freshness
### Follow-up Questions
- A major event makes hundreds of publishers post the same story within minutes. How do you keep the feed from filling up with near-identical items?
- A publisher's feed goes down or starts returning malformed data. What does the user see, and how does the system recover?
- How would you personalize ranking by reading history without slowing down the read path?
- How do you paginate a feed that receives new articles while the user is scrolling?
Overview: Design a news aggregator that collects articles from many publishers and serves users a feed of recent news. It tests traffic estimation, ingestion and deduplication, the core data structures behind the feed, database selection, and failure handling for every service in the pipeline.
Read the full Rippling Software Engineer interview experience this question came from