Make a Hostname-Limited Crawler Concurrent

A crawler receives a `startUrl` and an `htmlParser` dependency that discovers linked URLs. It should crawl only URLs reachable from the start under the same hostname. Remove each URL's fragment before deduplicating it. Explain a sequential approach, then extend it to concurrent execution. Focus on thread-safe discovery and preventing the same deduplicated URL from being fetched by multiple workers. State the parser, URL-resolution, and failure assumptions needed for the concurrent design. ### What a Strong Answer Covers - Hostname-based scope checks and fragment removal without unrelated URL rewrites. - Atomic ownership of a newly discovered URL before it is scheduled. - Coordination of queued and active work so completion is detected correctly. - Tests involving different hostnames, fragments, cycles, and concurrent discovery of the same page. ### Follow-up Questions - Why can a thread-safe set still be used incorrectly in a check-then-fetch sequence? - Why does an empty work queue not necessarily mean the crawl has finished?