Scale a file-crawling API using async jobs

Q: Scale a file-crawling API using async jobs

This question evaluates a candidate's ability to design scalable, fault-tolerant systems for long-running file-crawl operations, including async job orchestration, pagination/streaming, concurrency control, caching/deduplication, failure handling, and storage-abstraction considerations.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty System Design question, commonly asked during Technical Screen rounds at Dropbox.

Q: What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Dropbox during technical interviews.

Question

Loading...

You implemented a synchronous API that lists all files under a directory (recursive crawl). Now you need to scale it.

Part A — Scaling discussion (no code required)

How would you scale the “list all files under a path” API when:

The directory tree can be huge (millions/billions of files)
Crawls can take minutes+ and may time out
Many users may request crawls concurrently
The underlying storage could be local disks or network storage (e.g., NFS / object-store-like abstraction)

Discuss:

API shape (sync vs async)
Pagination/streaming of results
Concurrency limits and backpressure
Caching and deduplication
Failure handling and partial results

Part B — Async job implementation (with provided templates)

Based on your design in Part A, implement (or describe at a detailed interface level) an async crawl job:

Client submits rootPath and gets back a jobId .
Clients can poll job status and fetch results.
The job should be robust to failures, support retries, and avoid repeating work excessively.

Specify key components/APIs, e.g.:

POST /crawl-jobs → { jobId }
GET /crawl-jobs/{jobId} → status/progress
GET /crawl-jobs/{jobId}/results?cursor=... → paged results

State any assumptions (e.g., maximum result size, retention window, storage type).

Scale a file-crawling API using async jobs

Quick Overview

Part A — Scaling discussion (no code required)

Part B — Async job implementation (with provided templates)

Solution

Comments (0)