Design a Scalable and Polite Web Crawler
Company: Amazon
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Overview: Design a horizontally scalable web crawler with a durable frontier, host-level politeness, deduplication, parsing, and recrawl planning. Examine URL normalization, leases, redirects, retries, backpressure, robots policy, crawler traps, content fingerprints, failure recovery, and crawl-quality metrics.
