Implement hostname-restricted web crawler
Overview
This question evaluates skills in graph traversal and cycle detection, hostname-based filtering, URL parsing, and basic network error handling. Commonly asked in Coding & Algorithms interviews, it assesses algorithmic reasoning about traversal strategy and complexity (time and space) analysis while balancing conceptual understanding with practical implementation details.
Implement a single-threaded web crawler that, given a starting URL startUrl and an interface getUrls(url) that returns all hyperlinks on the page at url, returns the set of all unique pages reachable whose hostname exactly matches the hostname of startUrl. Requirements: do not fetch pages outside the starting hostname; avoid duplicate fetches and infinite loops (handle cycles); choose BFS or DFS and justify your choice; state time and space complexity; explain how you parse the hostname from a URL; describe basic error handling for failed requests.