Design a Concurrent Domain Crawler
Company: Anthropic
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Technical Screen
Design a crawler that starts from one seed URL and explores all reachable pages in the same domain efficiently.
Discuss:
- How you would structure the frontier of URLs to visit.
- How to prevent duplicate fetches across concurrent workers.
- Whether you would use asynchronous I/O, multithreading, or multiprocessing, and why.
- How to detect when the crawl is complete.
- How to handle failures, timeouts, and back-pressure.
- How to enforce politeness limits such as request throttling.
Assume the workload is primarily fetching web pages over the network.
Quick Answer: This question evaluates a candidate's understanding of concurrent system design, web crawling fundamentals, URL frontier organization, duplicate detection, concurrency control, network I/O models, fault tolerance, and politeness mechanisms such as throttling.