Design and implement a basic web crawler that fetches pages concurrently using a thread executor. Requirements: accept one or more seed URLs; use robust URL parsing to normalize and resolve links; avoid revisiting the same normalized URL; respect robots.txt and per-host politeness (rate limiting); cap concurrency and depth; optionally restrict to same-origin. Handle redirects, HTTP errors, timeouts, and content-type filtering. Describe data structures for the frontier and visited set, duplicate detection strategy, and how you would test and monitor the crawler.

This question evaluates understanding of concurrent system design, networking and parsing concerns, URL normalization and deduplication strategies, per-host politeness and rate limiting, error handling and observability for building robust web crawlers.

How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Onsite rounds at Anthropic.

What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Anthropic during technical interviews.

Design a concurrent web crawler | Anthropic Interview Question

System Design: Concurrent Web Crawler (Threads)

You are asked to design and implement a basic web crawler that fetches pages concurrently using a thread executor. The crawler should be production-conscious (correctness, robustness, and observability) while remaining reasonably simple.

Requirements

Input
1. Accept one or more seed URLs.
2. Optional flag to restrict crawling to the same origin as the seeds (scheme, host, port).
Crawling behavior
1. Fetch pages concurrently using a thread executor with a configurable max worker count.
2. Cap crawl depth from each seed.
3. Extract links from HTML pages and enqueue newly discovered URLs.
4. Normalize and resolve links robustly (relative links, fragments, default ports, casing, etc.).
5. Avoid revisiting the same normalized URL (dedup across in-queue and visited).
Compliance and politeness
1. Respect robots.txt (allow/disallow rules per user-agent; cache per host; honor crawl-delay if present).
2. Per-host politeness/rate limiting (e.g., at most 1 request per host per X seconds, configurable; honor Retry-After on 429/503).
Networking
1. Handle redirects (update to final URL; dedup on the normalized final URL).
2. Handle HTTP errors and timeouts gracefully (do not crash; backoff when appropriate).
3. Filter by content type (e.g., only text/html by default).
Data structures and strategy
1. Describe the frontier and visited set data structures.
2. Describe the duplicate detection strategy (including enqueued vs. fetched URLs and redirects).
Testing and monitoring
1. Explain how you would test the crawler (unit, integration, concurrency, and fault-injection tests).
2. Describe what you would monitor/measure in a real run (metrics, logs, alerts).

Deliverables

A brief architecture description and rationale.
Core algorithm and key components (pseudo-code or code sketch is fine).
Clear description of data structures and dedup logic.
Testing strategy and monitoring plan.

Requirements

Input

Accept one or more seed URLs.
Optional flag to restrict crawling to the same origin as the seeds (scheme, host, port).

Crawling behavior

Fetch pages concurrently using a thread executor with a configurable max worker count.
Cap crawl depth from each seed.
Extract links from HTML pages and enqueue newly discovered URLs.
Normalize and resolve links robustly (relative links, fragments, default ports, casing, etc.).
Avoid revisiting the same normalized URL (dedup across in-queue and visited).

Compliance and politeness

Respect robots.txt (allow/disallow rules per user-agent; cache per host; honor crawl-delay if present).
Per-host politeness/rate limiting (e.g., at most 1 request per host per X seconds, configurable; honor Retry-After on 429/503).

Networking

Handle redirects (update to final URL; dedup on the normalized final URL).
Handle HTTP errors and timeouts gracefully (do not crash; backoff when appropriate).
Filter by content type (e.g., only text/html by default).

Data structures and strategy

Describe the frontier and visited set data structures.
Describe the duplicate detection strategy (including enqueued vs. fetched URLs and redirects).

Testing and monitoring

Explain how you would test the crawler (unit, integration, concurrency, and fault-injection tests).
Describe what you would monitor/measure in a real run (metrics, logs, alerts).

Design a concurrent web crawler

Quick Overview

System Design: Concurrent Web Crawler (Threads)

Requirements

Deliverables

Solution

Comments (0)

Design a concurrent web crawler

Quick Overview

System Design: Concurrent Web Crawler (Threads)

Requirements

Deliverables

Solution

Comments (0)