This question evaluates system design skills for building a large-scale, multi-region news aggregation platform, assessing competencies in ingestion architecture, deduplication and clustering, indexing and retrieval, personalization and ranking, and operational concerns like monitoring, consistency, and compliance.
Design a news aggregation system similar to Google News. Emphasize the ingestion layer: publisher onboarding/authentication, RSS/sitemap/crawl scheduling, politeness and rate limiting, fetcher architecture, schema normalization and enrichment (language detection, category tagging), deduplication and near-duplicate clustering, near-real-time updates and backfill, idempotency and exactly-once semantics, retry strategies, spam/abuse filtering, copyright and robots compliance, and monitoring/alerting. Then cover storage and retrieval: article store, indexing, feed generation, personalization/ranking, freshness and caching. Specify APIs, data models, consistency guarantees, multi-region scalability/partitioning, SLAs, and provide capacity estimates.
Overview: This question evaluates system design skills for building a large-scale, multi-region news aggregation platform, assessing competencies in ingestion architecture, deduplication and clustering, indexing and retrieval, personalization and ranking, and operational concerns like monitoring, consistency, and compliance.
Design a Google News–like News Aggregation Platform
Design a multi-region news aggregation platform (in the spirit of Google News) that ingests content from many third-party publishers and serves near-real-time, deduplicated, categorized, and personalized news feeds to users on web and mobile.
The system must support both ingestion modes:
Pull-based
: crawling publisher RSS/Atom feeds and sitemaps on a schedule.
Push-based
: publishers notify us of new content via WebSub/webhooks.
The interviewer has signaled that the ingestion layer is the focus — go deep there. Storage, retrieval, and the cross-cutting concerns (APIs, data models, consistency, multi-region, SLAs, capacity) should be covered competently but at lighter depth.
Constraints & Assumptions
These are starting numbers to anchor your estimates; refine them with the interviewer.
~2M new or updated articles per day; expect ~10× bursts during major news events.
Average raw article HTML ~100 KB; normalized metadata JSON ~10 KB/article.
~5M daily active users issuing ~100M feed/search requests per day.
Freshness target
: an eligible article should appear in relevant feeds within minutes of publication (treat p95 ≈ 3 minutes as the goal).
Compliance is a hard constraint
, not a feature:
robots.txt
,
noarchive
/
nosnippet
, paywall/licensing rules, and DMCA takedowns must be honored end-to-end.
Multi-region with active-active reads; ingestion can be localized but story deduplication must be
global
.
Clarifying Questions to Ask Guidance
Do we store and display
full article text
, or
snippets + metadata only
? (This drives both storage cost and legal exposure, and varies per publisher's license.)
Is
search
a first-class product surface, or a secondary feature on top of feeds?
For v1, is
personalization
required, or is a strong popularity/recency ranking baseline sufficient?
What is the relative priority of
freshness vs. correctness of clustering
when they conflict (e.g., show a story fast vs. wait to collapse near-duplicates)?
Are there hard
data-residency
requirements (must certain publishers' content stay in-region)?
What is the acceptable end-to-end SLA for a
DMCA takedown
to disappear from serving (including caches/CDN)?
Part 1 — Ingestion Layer (primary focus)
Design the end-to-end ingestion pipeline that turns publisher content into clean, deduplicated, enriched, indexable articles. At minimum, address: publisher onboarding and authentication (domain verification, signing keys); source discovery and adaptive scheduling; politeness and rate limiting (robots.txt, per-host concurrency, backoff); fetcher architecture; schema normalization and enrichment (canonicalization, boilerplate extraction, language detection, category tagging, NER); deduplication and near-duplicate story clustering; near-real-time updates vs. historical backfill; idempotency and exactly-once semantics under retries/replays; a retry taxonomy (transient vs. permanent) with dead-letter handling; spam/abuse filtering; copyright/robots compliance; and monitoring/alerting/auditing.
What This Part Should Cover Guidance
Pipeline decomposition into independent, replayable stages; how each stage scales (queue-lag autoscaling) and isolates failures.
Idempotency/exactly-once
reasoning: deterministic IDs, idempotent upserts, and why effective exactly-once-to-readers is the achievable goal.
Politeness and a precise retry taxonomy
: per-host token buckets keyed on host+IP, conditional GET, backoff with jitter and circuit breakers, transient-vs-permanent handling, and DLQs.
Compliance
treated as a hard constraint inside the pipeline (rights flags gating storage/display,
noarchive
/paywall handling).
Part 2 — Storage & Retrieval
Design how articles are stored and how feeds and search are served. Cover: the article store (raw vs. normalized, versioning for corrections); indexing for search and feed candidate retrieval; feed generation for global / topic / locale / personalized surfaces; personalization and ranking; and freshness/caching.
What This Part Should Cover Guidance
Storage tier choices justified by access pattern, retention/TTL, and versioning for corrections vs. takedowns.
Search/feed
indexing
strategy (time + language sharding, filterable fields, NRT refresh with backpressure).
A clear
feed-generation recipe
including cluster-level dedup, diversity constraints, and pagination.
A
ranking
model that layers personalization over a popularity/recency/quality baseline, with a feature store and exploration.
Freshness/caching
with appropriate TTLs, stampede protection, and warmers.
Part 3 — Cross-Cutting Specifications
Pull the design together: define the key APIs (publisher/admin, ingestion webhook, consumer feeds/search), the data models (Publisher, SourceFeed, CrawlJob, Article, Cluster, UserProfile, Event), the consistency guarantees for each surface, the multi-region scalability/partitioning scheme, the SLAs/SLOs, and a back-of-the-envelope capacity estimate with a scaling plan.
What This Part Should Cover Guidance
A coherent, minimal
API surface
with the right auth per audience (OAuth2 for consumers; API key + HMAC/
Idempotency-Key
for publisher push).
Data models
whose keys reflect the partitioning and idempotency design (deterministic article/version ids, cluster ids).
Per-surface consistency
statements (ingestion effective-exactly-once; read-after-write where it matters; eventual elsewhere bounded by the freshness SLO).
A
multi-region partitioning
scheme (fetchers by host hash; storage/search sharded; active-active reads) and how
global dedup
is preserved.
A
capacity estimate
(ingest QPS/bandwidth, storage, serving RPS, cache hit rates) that drives the
scaling plan
.
What a Strong Answer Covers Guidance
Across all three parts, a strong candidate demonstrates the cross-cutting judgment that ties the design together:
Correctly identifies that ingest
QPS is modest but bursty and bandwidth-heavy
, so the real difficulty is correctness (dedup/clustering, idempotency, compliance), not raw throughput — and prioritizes accordingly.
Reasons explicitly about
consistency trade-offs
, choosing strong consistency only where it's cheap and necessary (article uniqueness) and eventual consistency on the hot path (story clustering), bounded by the freshness SLO.
Treats
compliance as a one-way door
: every takedown invalidates object store
and
index
and
CDN with an audit trail.
Connects estimates to architecture: numbers justify the fetcher pool split, autoscaling on queue lag, cache hit targets, and sharding strategy.
A breaking story is reported by 200 publishers within 90 seconds. Walk through exactly what happens in the dedup/clustering stage, and how you keep the feed from showing 200 near-identical headlines while still surfacing it within the freshness SLO.
A publisher issues a correction (updated body, same URL) every few minutes to game freshness ranking. How does your versioning + ranking distinguish a legitimate correction from republish-spam?
A top-20 publisher's ingest volume silently drops to zero. How does your monitoring detect this, and how do you tell a parser break from a legitimate news lull or a blocked crawler?
How would you evolve the ranking from a popularity baseline to true per-user personalization without tanking CTR during the transition?