# Design a Scalable and Polite Web Crawler Design a web crawler that begins from a set of seed URLs, discovers links, fetches eligible pages, and stores crawl results and metadata. The design should scale horizontally while respecting per-site politeness and avoiding uncontrolled duplicate work. ### Constraints & Assumptions - Pages, DNS, and network requests may fail, redirect, hang, or return unexpectedly large bodies. - Different URLs may refer to the same resource, while different resources may share identical content. - The crawler must obey an explicitly chosen robots and politeness policy. - Work is at least once, so repeated delivery or processing is possible. - Freshness requirements and crawl scope must be configurable rather than implicit. ### Clarifying Questions to Ask - Is the goal a one-time crawl, continuous recrawl, or both? - Which schemes, domains, content types, and authentication contexts are in scope? - What scale, freshness, and retention targets matter? - Is exact coverage more important than fast discovery? ### Part 1: Core Data Flow Describe the URL frontier, fetchers, parsers, deduplication, storage, and recrawl planning. #### What This Part Should Cover - URL normalization and scope checks before frontier insertion. - Host-aware scheduling, fetch limits, redirect handling, and bounded content parsing. - Separate URL-seen and content-fingerprint decisions. - Durable page metadata and discovered-link records. ### Part 2: Distributed Coordination Explain how work is partitioned, leased, retried, and recovered without violating host-level rate limits. #### What This Part Should Cover - Ownership keyed by host or another scheme that preserves politeness. - Durable leases or visibility timeouts and idempotent result writes. - Backoff, retry classification, poison-item handling, and backpressure. ### Part 3: Quality and Operations Describe tests, observability, and how you would evolve from an initial design to a larger deployment. #### What This Part Should Cover - Deterministic fixtures for redirects, loops, robots rules, malformed pages, and duplicate links. - Coverage, freshness, latency, queue, failure, and politeness metrics. - Capacity limits and protections against unbounded pages, link explosions, and crawler traps. ### What a Strong Answer Covers - A frontier that combines durability, deduplication, priority, and per-host scheduling. - Clear canonicalization limits and no assumption that URL equality equals content equality. - Safe at-least-once processing with leases, idempotency, and reconciliation. - Operational controls for politeness, failures, resource limits, recrawls, and scope. ### Follow-up Questions - How would you prioritize frequently changing or high-value pages? - What happens when robots policy changes after a URL is already queued? - How would you detect and contain an infinite calendar or faceted-navigation trap?