Design and implement a web crawler. First, build a single‑threaded version that, given a set of seed URLs, fetches pages, extracts links, deduplicates visits, respects robots.txt, and stops at a configurable depth and/or domain scope. Next, extend it to a concurrent version and compare three approaches: (a) manual multithreading using queues and locks, (b) a fixed‑size thread pool, and (c) an asyncio/event‑loop model. For each approach, explain and implement how you: maintain a URL frontier, enforce per‑host politeness and rate limits, avoid duplicate fetches, handle failures/retries and backoff, manage back‑pressure, and perform graceful shutdown. Discuss data structures (e.g., visited sets, frontier queues, per‑host buckets), synchronization primitives, and mechanisms to prevent deadlocks/starvation. Analyze time/space complexity and evaluate correctness, liveness, and performance trade‑offs across the three concurrency models.

This question evaluates a candidate's skills in concurrent system design, including concurrency primitives, synchronization, URL frontier management, per-host politeness and rate-limiting, fault tolerance, and trade-off analysis for a web crawler.

How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Technical Screen rounds at Anthropic.

What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Anthropic during technical interviews.

Design a concurrent web crawler | Anthropic Interview Question

System Design: Web Crawler (Single-threaded and Concurrent)

Context

Design and implement a web crawler that starts from a set of seed URLs and explores the web while respecting operational constraints (robots.txt, scope, depth). Then extend the design to three concurrent implementations and compare them.

Assume you can use Python and reasonable open-source libraries for HTTP and HTML parsing. Focus on correctness, liveness, and performance trade-offs. Show code-level designs (pseudo-code or real code skeletons) and justify data structures and synchronization choices.

Requirements

Single-threaded crawler
- Inputs: seed URLs, max depth, optional allowed domain scope.
- Fetch pages, extract links, deduplicate visits, respect robots.txt, stop by depth and/or scope.
- Handle failures/retries with backoff.
Concurrent crawler: compare three approaches a) Manual multithreading using queues and locks b) Fixed-size thread pool c) Asyncio/event-loop model For each approach, explain and implement how you:
- Maintain a URL frontier
- Enforce per-host politeness and rate limits
- Avoid duplicate fetches
- Handle failures/retries and backoff
- Manage back-pressure
- Perform graceful shutdown
Discuss
- Data structures (visited sets, frontier queues, per-host buckets)
- Synchronization primitives and how you prevent deadlocks/starvation
- Time/space complexity
- Correctness, liveness, and performance trade-offs across the three models

Context

Requirements

Single-threaded crawler

Inputs: seed URLs, max depth, optional allowed domain scope.
Fetch pages, extract links, deduplicate visits, respect robots.txt, stop by depth and/or scope.
Handle failures/retries with backoff.

Concurrent crawler: compare three approaches a) Manual multithreading using queues and locks b) Fixed-size thread pool c) Asyncio/event-loop model For each approach, explain and implement how you:

Maintain a URL frontier
Enforce per-host politeness and rate limits
Avoid duplicate fetches
Handle failures/retries and backoff
Manage back-pressure
Perform graceful shutdown

Discuss

Data structures (visited sets, frontier queues, per-host buckets)
Synchronization primitives and how you prevent deadlocks/starvation
Time/space complexity
Correctness, liveness, and performance trade-offs across the three models

Design a concurrent web crawler

Quick Overview

System Design: Web Crawler (Single-threaded and Concurrent)

Context

Requirements

Solution

Submit Your Answer

Design a concurrent web crawler

Quick Overview

System Design: Web Crawler (Single-threaded and Concurrent)

Context

Requirements

Solution

Submit Your Answer