Design News Pull Scheduling, Article Storage, and Topic Reads

Design a news aggregation service similar in purpose to Google News. Each article has a topic ID. Focus on pulling articles from sources, storing article information, and serving topic-based article results. ### Part 1 — Schedule pull workers Explain how source pulls are scheduled and how the system recovers when a worker or pull attempt fails. #### What This Part Should Cover - Durable scheduling state, work ownership, and retry behavior. - Prevention of lost pulls and duplicate article creation when work is retried. - Isolation of slow or failing sources from healthy ones. ### Part 2 — Store article information Describe the article data model and the indexes or read structures needed for topic-based reads. #### What This Part Should Cover - Article identity, source identity, topic ID, publication information, and content references. - Deduplication, updates, and a declared ordering and pagination policy. ### Part 3 — Protect serving latency Explain how the read path avoids waiting for source pulls and how you would keep latency within an agreed target. #### What This Part Should Cover - Bounded indexed reads and selective caching or materialization. - Freshness and invalidation behavior as article data changes. - Load measurements, overload handling, and recovery when a dependency is slow. ### What a Strong Answer Covers - A coherent path from scheduled pull through durable article storage to the served page. - Explicit treatment of at-least-once work and stale results. - No invented traffic figures, freshness targets, or claims about Google's internal architecture. ### Follow-up Questions - A worker stores new articles and crashes before acknowledging its pull. What happens on retry? - How would you paginate a topic whose newest articles keep changing during a user's session?