## Design a News Aggregation Service Design a service that ingests articles from many news sources, normalizes and groups related coverage, and lets users browse or search a fresh feed. Clarify source contracts, personalization, ranking, geography, supported languages, and freshness targets before selecting components. ### Part 1 — Define Sources, Stories, and APIs Specify source, fetched article, canonical article, story cluster, feed, and user-preference identities plus ingestion and read APIs. #### What This Part Should Cover - Stable source URL or publisher IDs and immutable fetched versions. - Separation of an article from a cluster representing one developing story. - Feed ordering, pagination, search, and source-attribution requirements. - Content rights, removal, correction, and user-personalization boundaries. ```hint Keep every publisher version Two articles can cover one event without becoming the same artifact, and one publisher can later correct its own text. ``` ### Part 2 — Ingest and Normalize Reliably Design polling, feeds or APIs, fetch scheduling, parsing, retries, source quotas, normalization, deduplication, and storage. #### What This Part Should Cover - Durable checkpoints and idempotent article-version writes. - Per-source politeness, backoff, and failure isolation. - Raw evidence retained separately from normalized fields. - Exact duplicate detection distinguished from probabilistic story clustering. ```hint Do not let clustering erase provenance A shared story ID can connect coverage while every displayed claim still links to its publisher and fetched version. ``` ### Part 3 — Build and Serve Feeds Design candidate retrieval, ranking, diversity, pagination, caching, and updates as a story develops. #### What This Part Should Cover - Explicit ranking inputs and a chronological fallback. - Deterministic cursor semantics under new arrivals and reranking. - Source and topic diversity without inventing unsupported user preferences. - Version-aware caches and bounded stale-result behavior. ```hint Freeze enough ordering context A page-number cursor cannot explain where to continue after newer articles arrive or scores change. ``` ### Part 4 — Correct, Remove, and Operate Handle corrections, article removal, cluster mistakes, source outages, reprocessing, abuse, and observability. #### What This Part Should Cover - Tombstones and version history rather than silent destructive updates. - Reversible cluster merge and split operations. - Replay from raw versions after parser or classifier changes. - Metrics for source lag, parse failures, duplicates, cluster quality, and feed freshness. ```hint Make a bad merge reversible Story clustering is uncertain derived state, so operators need to separate articles later without rewriting source history. ``` ### What a Strong Answer Covers - Preserves source versions and attribution throughout the pipeline. - Separates exact deduplication from uncertain story grouping. - Provides stable, fresh, and explainable feed reads. - Includes correction, removal, replay, source isolation, and quality monitoring. ### Follow-up Questions 1. How would you detect two publishers copying the same wire article? 2. What should happen when a source corrects a headline after publication? 3. How would you evaluate story clustering across languages? 4. How can feed pagination remain usable while ranking scores change?