##### Question Design and implement a concurrent web crawler. Starting from one or more seed URLs, the crawler should fetch pages, extract links, deduplicate visits, stay within a configured scope (a single domain, a set of allowed domains/subdomains, and/or a maximum link depth), and respect `robots.txt`. Assume the workload is primarily network-bound (most time is spent waiting on HTTP responses). Address the following sub-parts: 1. **Single-threaded baseline.** Build a single-threaded crawler that, given seed URLs, fetches pages, extracts and normalizes links, deduplicates visits, respects `robots.txt`, and stops at a configurable depth and/or domain scope. 2. **URL frontier.** How do you structure the frontier of URLs waiting to be visited so that ordering (e.g. BFS) and per-host fairness are preserved? 3. **Deduplication across concurrent workers.** How do you prevent duplicate fetches when many workers run at once? Cover URL normalization and why the `visited` check-and-insert must be effectively atomic and done at *schedule* time, not after fetching. 4. **Concurrency model: async I/O vs. multithreading vs. multiprocessing.** Discuss which model you would choose and why for an I/O-bound workload. Then extend the baseline into a concurrent version and compare three concrete implementations: - (a) manual multithreading using queues and locks, - (b) a fixed-size thread pool, and - (c) an asyncio / event-loop model. For each, explain how you maintain the frontier, enforce per-host politeness/rate limits, avoid duplicate fetches, handle failures/retries/backoff, manage back-pressure, and perform graceful shutdown. 5. **Completion detection.** How do you reliably detect that the crawl is finished (not merely that the queue momentarily emptied)? 6. **Failures, timeouts, retries, and back-pressure.** How do you handle transient vs. permanent errors, set timeouts, retry with backoff, and bound memory/concurrency under load? 7. **Politeness and rate limiting.** How do you enforce per-host throttling and `robots.txt` crawl-delay so you do not overload a target site? 8. **Analysis.** Discuss the relevant data structures (visited sets, frontier queues, per-host buckets) and synchronization primitives, how you prevent deadlocks/starvation, the time/space complexity, and the correctness/liveness/performance trade-offs across the three concurrency models.

An Anthropic software-engineering system-design screen asking you to design and implement a concurrent web crawler: a single-threaded baseline, then a concurrent version comparing manual multithreading, a fixed-size thread pool, and asyncio. It covers URL frontier design, deduplication, per-host politeness and rate limiting, failure/retry handling, back-pressure, completion detection, graceful shutdown, and complexity/trade-off analysis.

How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Technical Screen rounds at Anthropic.

What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Anthropic during technical interviews.

Design a concurrent web crawler | Anthropic Interview Question

Question

Design and implement a concurrent web crawler. Starting from one or more seed URLs, the crawler should fetch pages, extract links, deduplicate visits, stay within a configured scope (a single domain, a set of allowed domains/subdomains, and/or a maximum link depth), and respect robots.txt. Assume the workload is primarily network-bound (most time is spent waiting on HTTP responses).

Address the following sub-parts:

Single-threaded baseline. Build a single-threaded crawler that, given seed URLs, fetches pages, extracts and normalizes links, deduplicates visits, respects robots.txt , and stops at a configurable depth and/or domain scope.
URL frontier. How do you structure the frontier of URLs waiting to be visited so that ordering (e.g. BFS) and per-host fairness are preserved?
Deduplication across concurrent workers. How do you prevent duplicate fetches when many workers run at once? Cover URL normalization and why the visited check-and-insert must be effectively atomic and done at schedule time, not after fetching.
Concurrency model: async I/O vs. multithreading vs. multiprocessing. Discuss which model you would choose and why for an I/O-bound workload. Then extend the baseline into a concurrent version and compare three concrete implementations:
- (a) manual multithreading using queues and locks,
- (b) a fixed-size thread pool, and
- (c) an asyncio / event-loop model. For each, explain how you maintain the frontier, enforce per-host politeness/rate limits, avoid duplicate fetches, handle failures/retries/backoff, manage back-pressure, and perform graceful shutdown.
Completion detection. How do you reliably detect that the crawl is finished (not merely that the queue momentarily emptied)?
Failures, timeouts, retries, and back-pressure. How do you handle transient vs. permanent errors, set timeouts, retry with backoff, and bound memory/concurrency under load?
Politeness and rate limiting. How do you enforce per-host throttling and robots.txt crawl-delay so you do not overload a target site?
Analysis. Discuss the relevant data structures (visited sets, frontier queues, per-host buckets) and synchronization primitives, how you prevent deadlocks/starvation, the time/space complexity, and the correctness/liveness/performance trade-offs across the three concurrency models.

Question

Address the following sub-parts:

Single-threaded baseline. Build a single-threaded crawler that, given seed URLs, fetches pages, extracts and normalizes links, deduplicates visits, respects robots.txt , and stops at a configurable depth and/or domain scope.
URL frontier. How do you structure the frontier of URLs waiting to be visited so that ordering (e.g. BFS) and per-host fairness are preserved?
Deduplication across concurrent workers. How do you prevent duplicate fetches when many workers run at once? Cover URL normalization and why the visited check-and-insert must be effectively atomic and done at schedule time, not after fetching.
Concurrency model: async I/O vs. multithreading vs. multiprocessing. Discuss which model you would choose and why for an I/O-bound workload. Then extend the baseline into a concurrent version and compare three concrete implementations:
- (a) manual multithreading using queues and locks,
- (b) a fixed-size thread pool, and
- (c) an asyncio / event-loop model. For each, explain how you maintain the frontier, enforce per-host politeness/rate limits, avoid duplicate fetches, handle failures/retries/backoff, manage back-pressure, and perform graceful shutdown.
Completion detection. How do you reliably detect that the crawl is finished (not merely that the queue momentarily emptied)?
Failures, timeouts, retries, and back-pressure. How do you handle transient vs. permanent errors, set timeouts, retry with backoff, and bound memory/concurrency under load?
Politeness and rate limiting. How do you enforce per-host throttling and robots.txt crawl-delay so you do not overload a target site?
Analysis. Discuss the relevant data structures (visited sets, frontier queues, per-host buckets) and synchronization primitives, how you prevent deadlocks/starvation, the time/space complexity, and the correctness/liveness/performance trade-offs across the three concurrency models.

Design a concurrent web crawler

Quick Overview

Design a concurrent web crawler

Question

Submit Your Answer to Earn 20XP

Design a concurrent web crawler

Quick Overview

Design a concurrent web crawler

Question

Submit Your Answer to Earn 20XP