Crawl Same-Domain Links
Company: Anthropic
Role: Software Engineer
Category: Coding & Algorithms
Difficulty: hard
Interview Round: Technical Screen
Implement a Python function that crawls a website starting from a seed URL and returns all unique pages reachable within the same domain.
Assume a helper function `get_links(url) -> list[str]` is available and returns all hyperlinks found on that page.
Requirements:
- Start from a single input URL.
- Only follow links whose hostname matches the hostname of the seed URL.
- Do not visit the same URL more than once.
- Handle cycles safely.
- Continue until there are no more unseen same-domain links.
You may choose either BFS or DFS traversal. Be prepared to discuss how you would normalize URLs to reduce duplicate visits.
Quick Answer: This question evaluates understanding of web crawling and link traversal algorithms, including URL normalization, same-domain filtering, deduplication, and cycle-safe graph traversal.