Design an image crawler for unlimited URLs

title: Image crawler overview: "Crawl from any number of root URLs, at any depth. Durable frontier, per-host politeness, one copy of each image." status: draft review: reliability: status: pending security: status: pending architecture: status: pending product: status: pending Image crawler What is already decided Thesis. A URL is a row in a durable frontier. Fetchers lease rows. They do not keep the queue in memory. Image bytes are stored once, under the hash of the bytes. Politeness is the next time a host may be fetched, not a sleep buried in a worker. | Topic | Decision | | --- | --- | | How big is a crawl? | No cap on seed count. No cap on depth. The fetch rate per host is capped | | What is stored? | Image bytes in S3. Crawl state and image metadata in Aurora Postgres. HTML is parsed and dropped | | Same URL again? | One row per canonical URL. The same handler skips a refetch for 7 days when its artifact is already stored | | Same bytes, two URLs? | One object. Both URL rows point at that hash | | Politeness | Per host. Default 1 request per second, 2 in flight. robots.txt can raise the delay | | Crash | A lease expires. The row returns to the frontier. Five failures and the URL is dead | | Where is the queue? | The queued row in urls. The lease is the dequeue. §4 | | What is the scheduler? | 64 ECS tasks, one per hash(host_id) partition. Each task leases, then a fetch worker in that same task opens the socket. §2 | | How we see it, and what we do | §7. Metrics say the