Design a Concurrent Domain Crawler

Q: Design a Concurrent Domain Crawler

This question evaluates a candidate's understanding of concurrent system design, web crawling fundamentals, URL frontier organization, duplicate detection, concurrency control, network I/O models, fault tolerance, and politeness mechanisms such as throttling.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Question

Design a crawler that starts from one seed URL and explores all reachable pages in the same domain efficiently.

Discuss:

How you would structure the frontier of URLs to visit.
How to prevent duplicate fetches across concurrent workers.
Whether you would use asynchronous I/O, multithreading, or multiprocessing, and why.
How to detect when the crawl is complete.
How to handle failures, timeouts, and back-pressure.
How to enforce politeness limits such as request throttling.

Assume the workload is primarily fetching web pages over the network.

Design a Concurrent Domain Crawler

Quick Overview

Solution

Comments (0)