Design crawler storing only image URLs
Company: Atlassian
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
Design a web crawler that extracts and stores only image URLs from HTML pages (e.g., <img src>, <source srcset>, CSS background-image within inline styles) but does not store full HTML bodies. Cover:
1) High-level architecture (URL frontier, fetchers, parsers, deduplication, storage, indexing, and a control plane).
2) Crawl politeness and compliance (robots.txt, per-host rate limiting, retries/backoff, user-agent, canonicalization, URL normalization, avoiding traps).
3) Parsing at scale (streaming parsers, charset handling, content-type verification, managing redirects).
4) Deduplication strategies (normalized URL keys, hash-based dedupe of image content or headers, handling srcset and relative URLs).
5) Storage design and schema for images and page-image relationships; propose DB choices (e.g., key-value for frontier, document/column store for metadata, object store for images if you later choose to fetch binary for validation).
6) Query and API design: endpoints to list images by domain, by crawl time, by MIME type; pagination and filters.
7) Sharding and scaling (per-host queues, consistent hashing, horizontal scaling of fetchers/parsers).
8) Fault tolerance and idempotency (at-least-once fetching, de-dup on write, replay safety).
9) Monitoring, metrics, and alerts (crawl rate, error codes, robots denials, queue depth, unique image URL rate).
10) Capacity planning assumptions and rough sizing; discuss data retention and privacy considerations.
Overview: This question evaluates proficiency in designing scalable, fault-tolerant web crawlers and related competencies such as HTML parsing, URL normalization, deduplication, storage and indexing, and query API design, and it belongs to the System Design domain.
Community answers
Answer by freny07pethelis1996
title: Image crawler
overview: "Crawl from any number of root URLs, at any depth. Durable frontier, per-host politeness, one copy of each image."
status: draft
review:
reliability:
status: pending
security:
status: pending
architecture:
status: pending
product:
status: pending
Image crawler
What is already decided
Thesis. A URL is a row in a durable frontier. Fetchers lease rows. They do not keep the queue in memory. Image bytes are stored once, under the hash of the bytes. Politeness is the next time a host may be fetched, not a sleep buried in a worker.
| Topic | Decision |
| --- | --- |
| How big is a crawl? | No cap on seed count. No cap on depth. The fetch rate per host is capped |
| What is stored? | Image bytes in S3. Crawl state and image metadata in Aurora Postgres. HTML is parsed and dropped |
| Same URL again? | One row per canonical URL. The same handler skips a refetch for 7 days when its artifact is already stored |
| Same bytes, two URLs? | One object. Both URL rows point at that hash |
| Politeness | Per host. Default 1 request per second, 2 in flight. robots.txt can raise the delay |
| Crash | A lease expires. The row returns to the frontier. Five failures and the URL is dead |
| Where is the queue? | The queued row in urls. The lease is the dequeue. §4 |
| What is the scheduler? | 64 ECS tasks, one per hash(host_id) partition. Each task leases, then a fetch worker in that same task opens the socket. §2 |
| How we see it, and what we do | §7. Metrics say the