Design a distributed web crawler
Company: Anthropic
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
Overview: This question evaluates the ability to design scalable, fault-tolerant distributed systems for web crawling, covering competencies such as URL deduplication, politeness (rate limiting and robots.