System Design: Web Crawler

Introduction

A web crawler fetches web content, starting with a set of seed URLs and following discovered links. It saves documents and fetch metadata for later processing. The design must decide both what to fetch next and when a host can safely receive another request.

Crawlers fetch web pages, parse content, and extract URLs for further crawling. A search engine uses this output for several kinds of processing:

• Data cleaning

• Indexing

• Relevance scoring (e.g., PageRank)

• Discovery feedback to the URL frontier, which remains part of the crawler

• Analytics

This lesson focuses on the crawler’s System Design, excluding downstream stages like indexing or ranking. For those, refer to the chapter on distributed search.