## Question Design a distributed system that archives all Wikipedia pages and keeps the archive current. You have one reliable primary machine, about one thousand unreliable and untrusted worker bots with different IP addresses, and persistent blob storage. Focus on discovery, deduplication, task assignment, worker failure, safe storage, and periodic recrawling rather than a reader-facing website. ### Constraints & Assumptions - Workers may disappear, retry, lie about completion, or return malformed content. - The primary coordinates metadata but cannot proxy every page body. - Pages link to other pages, and the frontier is much larger than memory. - Fetches must respect site policy, pacing, and robots requirements. - An archived version is published only after content and provenance checks pass. ### Clarifying Questions to Ask - Are historical versions required, or only the latest snapshot? Preserve versioned snapshots when content changes. - May official dumps or APIs be used? Prefer them where allowed; crawling fills freshness or coverage gaps. - What freshness target and total bandwidth budget apply? - How is worker trust established, and which results require independent verification? - Are media assets in scope? Treat them as a separate typed crawl with its own policy. ```hint Make the primary a lease coordinator Store durable URL state and issue bounded leases. A missing heartbeat returns work to the frontier without trusting a worker-local queue. ``` ```hint Verify before publication Workers should upload immutable content blobs and signed result metadata; the coordinator validates URL, status, hash, type, and consistency before advancing page state. ``` ### What a Strong Answer Covers - Canonical URL discovery, durable frontier states, deduplication, and crawl politeness. - Expiring leases, idempotent result submission, retry budgets, and dead-letter handling. - Defenses against untrusted workers, including validation, sandboxing, sampling, and redundant fetches for suspicious results. - Content-addressed blob storage, manifests, version history, and atomic publication. - Recrawl scheduling based on change history, conditional requests, and freshness goals. - Primary recovery, metadata backups, observability, and resource controls. ### Follow-up Questions 1. How would you stop one malicious bot from poisoning the archive? 2. How can the system avoid crawling duplicate URL variants? 3. What state must survive loss of the primary machine? 4. How would official database dumps change the architecture?