Design a Large-Scale Polite Web Crawler
Company: Google
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Overview: Design a polite distributed web crawler for roughly ten billion pages and periodic recrawls. Cover a host-aware frontier, robots enforcement, deduplication, traps, durable ownership, recovery, DNS, and 24-hour capacity bottlenecks.
Read the full Google Software Engineer interview experience this question came from