Design an image crawler for unlimited URLs

Q: Design an image crawler for unlimited URLs

This question evaluates a candidate's ability to design scalable, fault-tolerant distributed systems for web crawling, covering competencies in concurrency, queueing and scheduling, deduplication, storage architecture, and observability.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Question

Design a service that crawls images starting from a set of root URLs.

Requirements:

Input: one or more root URLs.
Crawl pages, discover links, and download image resources.
Support unlimited number of root URLs and unlimited crawl depth .
Must handle failures (network errors, timeouts, crashes) and avoid re-crawling the same URL excessively.
Discuss storage for downloaded images and metadata.

Deliverables:

High-level architecture (components, data flow).
Queue/scheduler design and politeness (per-host rate limiting).
Deduplication strategy.
DB schema for crawl state and results.
Failure/retry model and monitoring.

Design an image crawler for unlimited URLs

Quick Overview

Solution

Comments (0)