Design a recursive distributed file crawler

Design a distributed service that crawls a large file system starting from a root path. A client should be able to call an API to start a crawl job, and background workers should traverse directories and files asynchronously. While processing a directory, a worker may split the work into smaller tasks and enqueue additional async jobs, so the async service can recursively trigger more of its own jobs. Discuss: - APIs for creating a crawl job and checking job status - the data model for crawl jobs, crawl tasks, and discovered file metadata - how workers recursively schedule child tasks - how to handle retries, idempotency, deduplication, and partial failures - how to scale to very large directory trees - the role of the database, message queue, and async workers - how to expose progress and final results to clients