Design a large-scale web crawler
Company: Meta
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Design a web crawler that continuously discovers and downloads web pages.
Requirements:
- Start from seed URLs and crawl at scale.
- Avoid duplicate crawling; obey robots.txt and per-domain rate limits.
- Prioritize important/fresh pages.
- Store fetched content and metadata.
- Be fault tolerant and resumable.
Out of scope: full search ranking, rendering heavy JS.
Overview: This question evaluates a candidate's ability to design scalable, fault-tolerant distributed systems for web-scale data acquisition, covering competencies such as crawler architecture, URL frontier management, deduplication, per-domain rate limiting and robots.