Design Distributed Storage for Large Structured Data

Design a distributed filesystem or storage service for large structured datasets, similar in purpose to a distributed data-processing filesystem. Go beyond a component diagram: explain concrete flows for metadata and data access, caching at different layers, job queues, node failures, and load balancing. ### Constraints & Assumptions - Clarify whether files are immutable, append-only, or randomly mutable, and how compute jobs consume them. - Separate namespace metadata from large data blocks. - No node count, data size, or latency target is supplied. State design choices without inventing deployment facts. ### Clarifying Questions to Ask - Is the workload dominated by large sequential scans, appends, or small random reads? - What durability and availability are required when a node or rack fails? - Are processing jobs part of this service or clients of it, and how important is data locality? - Which cached data may be stale, and how is a file version identified? ### Part 1 — Storage and cache flows Describe writing and reading a large file, including block placement, metadata publication, and cache use. #### What This Part Should Cover - Block IDs, file versions/manifests, metadata authority, and durable write acknowledgement. - Client metadata caches, block/data caches, and their invalidation or versioning rules. - Locality and bounded read amplification for structured data. ### Part 2 — Node failure and recovery Describe detection, replica selection, repair, and metadata availability. #### What This Part Should Cover - Checksums, failure detection, placement across failure domains, and repair prioritization. - A clear authoritative metadata state under failover. - Recovery from interrupted writes without publishing incomplete files. ### Part 3 — Jobs and load balancing Explain how queued work is assigned and how storage/compute resources are balanced. #### What This Part Should Cover - Job identity, attempts, retries, locality-aware scheduling, and output commit rules. - Capacity-aware placement and rebalancing with bandwidth limits. - Backpressure and metrics separating metadata, disk, network, and worker bottlenecks. ```hint Separate safety from repair speed A missing replica requires repair, but repair traffic can overload the same disks and network serving healthy reads. Define both the durability priority and the resource budget. ``` ### What a Strong Answer Covers - Concrete request and failure sequences for an infrastructure storage system. - Consistent versioning, cache, replica, and job semantics. - Trade-offs grounded in the stated access pattern and failure model. ### Follow-up Questions - How would you avoid stale cached block locations after a storage node fails? - What happens if a job finishes writing output but loses its execution lease before commit?