Design a concurrent web crawler

Q: Design a concurrent web crawler

This question evaluates skills in concurrent systems and orchestration, including thread-safe deduplication, URL normalization, global and per-host rate limiting, robots.txt compliance, error handling, and scalability trade-offs.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Onsite rounds at Snowflake.

Q: What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Snowflake during technical interviews.

Question

Web Crawler System Design (Onsite)

Problem

Design and implement a concurrent web crawler that:

Starts from a given URL.
Uses a provided interface to fetch outgoing links from a page.
Returns all pages under the same hostname as the starting URL.

Requirements

Do not revisit URLs; handle cycles safely.
Enforce a configurable global concurrency limit.
Ensure thread-safe deduplication.
Normalize URLs consistently before comparison/deduplication.
Be polite:
- Respect robots.txt rules for a given user-agent.
- Enforce rate limiting per host; consider Crawl-delay if present.
Robust error handling and retry policy.
Explain how you would test correctness and performance.

Given Interface (Assumed)

fetchOutgoingLinks(url: string) -> List[string]
- Returns absolute or relative URLs found on the page at url .
- May throw transient or permanent errors.

Assumptions

"Same hostname" means exact match of the host portion (no subdomains).
Both http and https may exist; treat them as distinct URLs, but only crawl those whose hostname matches the start URL's hostname.
Content parsing is handled by fetchOutgoingLinks ; your crawler focuses on orchestration, deduplication, and policy.
A simplified, single-process design is sufficient (discuss how to extend/distribute if time permits).

Design a concurrent web crawler

Quick Overview

Design a concurrent web crawler

Web Crawler System Design (Onsite)

Problem

Requirements

Given Interface (Assumed)

Assumptions

Submit Your Answer to Earn 20XP