This question evaluates understanding of scalable web crawler architecture, distributed systems concepts, URL and content deduplication, scheduling and prioritization, storage and metadata design, and operational concerns such as politeness, rate limiting, DNS/connection management, and fault tolerance.
Design a scalable web crawler that discovers and downloads web pages across the public internet. Specify the architecture (URL frontier, fetchers, parsers, storage), how you respect robots.txt and crawl-delay, how you deduplicate URLs and content, and how you prioritize and schedule crawling. Follow-up: extend the crawler to use multithreading and/or multiple machines—explain concurrency controls, per-host rate limiting, back-pressure, fault tolerance, and how you ensure exactly-once or at-least-once processing.
Quick Answer: This question evaluates understanding of scalable web crawler architecture, distributed systems concepts, URL and content deduplication, scheduling and prioritization, storage and metadata design, and operational concerns such as politeness, rate limiting, DNS/connection management, and fault tolerance.
Design a production-ready web crawler that discovers and downloads publicly accessible web pages at internet scale. The system starts from a list of seed URLs and must continuously discover, download, and recrawl pages over time while staying polite toward publishers (respecting robots.txt and crawl rate), avoiding duplicate work and storage, sidestepping crawler traps, and sustaining high throughput.
Your design should be modular: the same core components should run on a single machine for a small job and fan out to a distributed cluster for billions of pages. Concretely, cover:
Architecture
— the core components (URL frontier, fetchers, parsers, storage, metadata/coordination) and how they connect, including DNS resolution, connection management, and content-type handling.
Robots & politeness
— fetching/caching
robots.txt
, obeying user-agent rules and crawl-delay, and per-host / per-domain rate limiting and connection concurrency.
Deduplication
— URL dedup via canonicalization plus a global "seen" structure, and content dedup for exact and near-duplicate pages.
Prioritization & scheduling
— which URL to fetch next (depth, quality, freshness, domain budgets) and how recrawls are scheduled for freshness.
Storage & metadata
— where raw content (blobs) and structured metadata (fetch status, fingerprints, link graph, robots cache) live.
Scale
— make reasonable assumptions and let them drive your component choices.
Constraints & Assumptions
Treat this as a
general-purpose, search-engine-style crawler
: continuously discover and download public pages, keep them reasonably fresh, and feed downstream consumers (indexer, link graph, archive). Building the index/ranking and the query-serving path are
out of scope
.
Plan for a corpus on the order of
billions of pages
with continual recrawl. Use the prompt's targets as a starting point — e.g. an initial seed set of ~
108
URLs and a target sustained fetch rate of ~
104
fetches/sec — and
derive
the rest (daily volume, bandwidth, storage, metadata size) from there.
Politeness is a hard constraint
, not a nice-to-have. Only public pages are crawled, and
robots.txt
/
noindex
are obeyed.
Workers and machines fail routinely; the crawl must make
monotonic progress
without losing URLs or double-fetching.
The dominant costs are
bandwidth, storage, and JS rendering
— call them out and design to bound them.
Clarifying Questions to Ask Guidance
What is the crawler
for
? A search index, freshness/news monitoring, an archive, and a security scanner share machinery but prioritize coverage vs. freshness vs. fidelity differently.
How fresh must content be, and is there a recrawl SLA (e.g. news in minutes vs. static docs in weeks)?
What is the target corpus size and sustained fetch rate, and what bandwidth/storage budget am I designing against?
Do we need to render JavaScript-heavy pages (headless browser), or is fetching the raw HTML sufficient for the common case?
What downstream consumers read our output, and in what format (raw WARC blobs, parsed text, the extracted link graph)?
Are there politeness or legal requirements beyond
robots.txt
(allowlist/denylist of domains, regional restrictions, a published bot identity/contact)?
What a Strong Answer Covers Guidance
Sizing that drives design.
Derives daily page volume, sustained bandwidth (~GB/s), raw + compressed storage/day, metadata-row footprint, and "seen"-set size from the stated targets — and uses those numbers to justify component choices (sharded NoSQL vs. single DB, object store, in-RAM Bloom filter).
A clear, decoupled architecture.
Names the core components (ingest, frontier, robots/politeness, DNS, fetchers, parser/extractor, dedup, content store, metadata store, coordinator, observability), connects them with durable queues / a message bus, and identifies the
two stateful "brains"
(frontier + metadata store) vs. the stateless workers.
URL canonicalization + a two-tier "seen" gate.
Deterministic canonicalization (scheme/host casing, default ports, fragments, path normalization, query-param sorting, tracking-param stripping) producing a stable fingerprint; a Bloom/cuckoo front gate backed by an authoritative compare-and-set in the metadata store — and a correct account of Bloom false-positive vs. false-negative behavior so no real URL is ever silently dropped.
Content dedup.
Exact via
sha256(body)
with reference-counted blobs; near-duplicate via SimHash/MinHash bucketed with LSH so only plausibly-similar docs are compared.
Robots & politeness as first-class infrastructure.
Fetch/parse/cache
robots.txt
with conditional GETs and a TTL; sensible fail-open (4xx) vs. fail-closed (5xx/timeout) behavior; per-host token buckets, per-IP limiting for shared hosting/CDNs, per-host connection caps, and adaptive backoff on
429
/
503
/
Retry-After
.
Frontier, prioritization & recrawl.
The two-level front-queue (priority) / back-queue (per-host) / host-scheduler-heap (timing) structure; a priority score combining freshness/change-rate, link importance, depth penalty, and per-domain budgets; and change-rate-driven recrawl scheduling clamped to sane bounds.
Fetching, parsing & trap avoidance.
Async cached DNS, timeouts/redirect/body caps, an identifying User-Agent, content-type handling, outlink extraction through canonicalize→robots→dedup, and cheap heuristics against infinite calendars, repeating path segments, session-id explosions, and faceted-nav cartesian products.
Storage & metadata model.
Append-only compressed (optionally WARC) blobs keyed by content fingerprint; a sharded wide-column/KV metadata store partitioned by URL id holding the URL
state machine
, host state, and robots cache.
Operational maturity.
Names the real bottleneck (host availability, not connections), the key tradeoffs (Bloom sizing, in-RAM vs. durable frontier, coverage vs. quality, SimHash vs. MinHash), and the observability/alerting needed to catch politeness breaches and queue lag.
Follow-up Questions Guidance
Concurrency & distribution.
Extend the single-node design to multithreaded and multi-machine operation. How do you shard work so per-host rate limiting stays correct, control concurrency and apply
back-pressure
when parsing/storage lags, and recover from worker crashes without losing or double-fetching URLs?
Processing semantics.
Can you guarantee
exactly-once
processing across an external HTTP fetch plus blob and metadata writes? If not, how do
at-least-once delivery + idempotent effects
(CAS leases, content-addressed blobs, version-checked metadata upserts) give you
effectively-once
outcomes?
Per-IP politeness across unrelated domains.
Host-based sharding makes per-host limiting local, but two unrelated sites can share one shared-hosting IP and land on different workers. How do you enforce a per-IP cap without putting a distributed lock on the hot path?
Freshness under a fixed budget.
If bandwidth caps total fetches/day below what full recrawl coverage needs, how do you allocate the budget between discovering new pages and recrawling known ones, and how do you decide per-page recrawl frequency?