Design a single- and multi-threaded web crawler

Web Crawler (single-threaded, then multi-threaded)

You are given:

• A starting URL startUrl (e.g., "http://news.example.com/a/index.html" ).
• An interface HtmlParser with a method getUrls(url) that returns all URLs found on the page at url .

Your task is to crawl web pages starting from startUrl and return all unique pages that are reachable by following links, subject to the rules below.

Rules

1. Same-hostname only: Only crawl pages whose hostname is the same as the hostname of startUrl .
   • The hostname is the part between the scheme ( http:// or https:// ) and the next / .
2. Fragment handling: URLs may contain a fragment part starting with # (e.g., http://a.com/x#section2 ).
   • You must strip the fragment (remove # and everything after it) before:
     • deciding whether you have already visited the URL (deduplication), and
     • including the URL in the returned result.
   • Example: http://a.com/x#p1 and http://a.com/x#p2 should be treated as the same page : http://a.com/x .
3. No other normalization: Do not perform other URL normalization (e.g., do not canonicalize trailing slashes, default ports, query parameter ordering, etc.). Only strip the fragment.

Part A — Single-threaded

Implement a single-threaded crawler that returns the set/list of visited URLs (after stripping fragments), restricted to the same hostname.

Part B — Follow-up: Multi-threaded

Now implement a multi-threaded crawler to speed up crawling.

• Multiple worker threads may call HtmlParser.getUrls concurrently.
• Your solution must be thread-safe and must not crawl the same sanitized URL more than once.

Output

Return all unique sanitized URLs that are reachable from startUrl and satisfy the hostname constraint.

Assumptions / Notes

• HtmlParser.getUrls(url) is a black box and may be slow (network/IO-like).
• The link graph may contain cycles.
• URLs returned by getUrls are absolute URLs.