## Interview Prompt Given blocking `fetch(url)` and `parse(page)` functions, design a concurrent web crawler that starts from seed URLs and terminates after exhausting all reachable in-scope pages. The crawler must deduplicate each normalized URL into one logical work item, allow only policy-bounded retries, bound concurrency, handle failures, respect scope and politeness rules, and know when no queued or in-flight work remains. ### Constraints & Assumptions - The reachable in-scope graph is finite but contains cycles and duplicate links. - Fetch can time out or fail transiently. - Worker count and every in-memory queue are bounded; a disk-backed frontier and temporary parse spools are allowed. - Only URLs matching the configured host or scope policy are admitted. - The crawler must not impose a per-page link cap or discard a reachable in-scope URL merely because an in-memory queue is full. ### Clarifying Questions to Ask - What URL normalization rules define duplicates? - How many retries and what per-host request rate are allowed? - May the frontier spill to local durable storage, and must crawl state survive process failure? ### What a Strong Answer Covers - One transactional admission step that deduplicates a URL and records it as ready work, so a failed bounded-queue offer cannot strand a URL already marked seen. - A coordinator-owned disk-backed frontier, bounded worker queue, and result protocol in which workers never recursively block while offering discovered URLs to the same full work queue. - Correct termination detection across both queued and active work. - Retry classification, backoff, terminal failures, and observability. - Lossless streaming or spooling for large parse results, with no arbitrary link cap and no locks held during network calls. ### Follow-up Questions - How would you distribute the crawler across several machines? - What if parsing discovers millions of links from one page? - How would you resume without refetching everything after a crash?