Design a Google News-style **personalized news aggregator**. The system continuously collects articles from many third-party publishers by **polling their APIs** (assume most publishers expose a REST/JSON feed and do *not* offer webhooks or streaming). Users can: - Browse a feed of current news. - **Follow** specific publishers. - **Select preferred content categories** (e.g. Business, Sports, Technology, Entertainment). Personalization is **deliberately limited to two explicit signals only**: the publishers a user follows and the categories they select. There is no implicit/behavioral personalization (no click-history models, no embeddings of the user). Design the end-to-end system, covering ingestion, storage, feed generation, and operations. ```hint Where to start This system has two activities with very different shapes: continuously pulling articles from thousands of third-party APIs, and answering a feed request fast. Ask yourself which one must be low-latency and which can run in the background — and what that implies about how you'd separate them and what sits between them. ``` ```hint What does "personalized" actually require here? The only personalization signals are *followed publishers* and *selected categories* — no ML, no per-user models. Given that, ask: how much of the work to answer a feed request is genuinely specific to *one* user, versus the same for everyone who follows a given publisher or picks a given category? Let that distinction drive where you spend compute and what you cache. ``` ```hint Where will the interviewer push? Two parts of this problem are deceptively hard. First, *when* and *how* do you poll thousands of APIs without either wasting requests or missing fresh stories — what state would you keep per publisher to do better than a fixed timer? Second, the same story is filed by many outlets — what does "the same article" even mean, and is one notion of sameness enough? Have a concrete plan for each before going deep on the rest. ``` ```hint Polling re-fetches the same article Polling is inherently at-least-once: overlapping polls and retries will hand you the same article more than once. Before you store anything, ask what property each ingestion stage needs so that seeing an article twice is harmless — and what identity you'd key that on. ``` ### Constraints & Assumptions State your assumptions explicitly; reasonable numbers are fine. A typical framing: - **Publishers:** ~$10{,}000$ sources, each polled on its own schedule; per-publisher rate limits and SLAs vary. - **Article volume:** on the order of $1$–$5$ million new/updated articles per day (tens of articles/sec average, with spikes around major events). - **Users:** ~$100$M registered, ~$10$M daily active; read-heavy ($\gg 100{:}1$ read:write at the feed layer). - **Feed latency target:** $p99 < 200$ ms for a cached first page; freshness target: a newly published article should be eligible for feeds within **a few minutes** (bounded by publisher polling cadence and rate limits). - **Availability:** feed serving should stay up and degrade gracefully if some publishers or the ranking layer fail; ingestion may lag without breaking reads. - Personalization signals are explicit only (followed publishers + selected categories) — no behavioral/ML personalization in scope. ### Clarifying Questions to Ask - How many publishers, and do any support webhooks/push or conditional requests (ETag, cursors), or is it pure polling? - What freshness SLA matters — must breaking news surface in seconds, or are minutes acceptable? - Read/write ratio and DAU scale — does the feed need precomputation, or is compute-on-read sufficient? - Is full-text keyword search of articles in scope, or only category/publisher filtering? - How aggressive should deduplication be — exact duplicates only, or near-duplicate clustering of the same story across publishers? - Are personalization signals truly limited to follows + categories, or should the design leave room for future engagement-based ranking? ### What a Strong Answer Covers - **Requirements discipline:** crisp functional vs. non-functional split; explicit scale estimates and SLAs; identifies that the read path is the latency-critical one. - **Adaptive polling:** per-publisher state (last poll, cursor, error count, rate-limit reset), conditional requests, frequency tuned by update rate/priority, backoff + jitter, respecting rate limits. - **At-least-once ingestion with idempotent processing** keyed on a stable article identity. - **Clean component decomposition** with an async queue between polling and processing; per-publisher fault isolation. - **Core APIs** for clients (feed, article detail, follow/unfollow, category preferences) and for internal ingestion (article upsert, publisher poll-state), with cursor-based feed pagination. - **Deduplication in stages:** exact (publisher article ID / canonical URL), content fingerprint, and near-duplicate clustering; choosing a representative article per cluster. - **Feed model:** candidate retrieval from followed publishers + selected categories, a transparent rule-based ranking formula (freshness, follow boost, category boost, diversity), and a clear serving strategy (compute-on-read with shared per-publisher/per-category caches vs. precompute). - **Caching + pagination:** short TTL on personalized pages, longer on article detail, cursor-based pagination with a stable sort key. - **Data model + storage choices** justified by access pattern (relational for users/follows/config; scalable KV/document store for article metadata; search index; cache). - **Operations:** freshness/lag/error monitoring, alerting thresholds, dead-letter handling, graceful degradation, and explicit tradeoffs (polling vs. push, compute-on-read vs. precompute). ### Follow-up Questions - A breaking story is published by 50 outlets within 2 minutes. Walk through exactly how your dedup pipeline clusters them and picks the one article a user sees — and what happens to the score of the cluster. - A single popular publisher's API starts returning `429`/`500`. How does your system protect freshness for *everyone else* and recover that publisher without manual intervention? - The product team now wants to add engagement-based ranking (clicks, dwell time) on top of follows/categories. What in your design has to change, and what stays the same? - You're told the $p99$ feed latency is fine but the *freshness* p99 is 8 minutes and the SLA is 2 minutes. Where do you look first, and which knobs do you turn?