Distributed Storage Architecture

What's being tested

Interviewers probe your ability to design a reliable, scalable, and maintainable distributed storage architecture for application workloads: choosing partitioning, replication, consistency, durability, and storage engine tradeoffs. They want to see system decomposition, latency/throughput calculations, failure-mode reasoning, and clear operational requirements (SLAs, workload shape). At eBay this maps to building backend services that store catalog, listings, and transaction metadata with predictable p99 latency and data durability guarantees.

Core knowledge

Partitioning: range vs consistent hashing; consistent hashing reduces re-sharding pain and hot-spotting; quantify: throughput roughly scales with partition count $P$ until per-node CPU or I/O becomes the bottleneck.
Replication strategies: synchronous (strong consistency) vs asynchronous (higher throughput, potential data loss); use quorum rules $R + W > N$ where $N$ is replicas, $R$ read quorum, $W$ write quorum.
Consensus algorithms: Raft and Paxos for leader-based strong-consistency; leader commit latency = leader election + majority ack; expect extra RTT per write for durability in leader-sync modes.
Storage engines: LSM-tree (RocksDB/LevelDB) for write-heavy workloads and high write amplification during compaction; B-tree for random-read heavy, lower write amplification.
Compaction & tombstones: garbage collection cost can spike; tombstones (deletes) require careful GC windows to avoid resurrecting deletes in eventual-consistency setups.
Consistency models: strong/linearizable, sequential, causal, and eventual; read-after-write guarantees require leader reads or appropriate quorum choices; state what you’ll provide and how clients should be informed.
Data durability and failure modes: durability probability increases with replication factor $N$ but costs $O(N)$ storage; consider erasure coding for large-object cost-efficiency vs CPU/repair complexity.
Rebalancing and data movement: moving a partition costs network I/O; estimate time = partition_size / available_replication_bandwidth and plan throttling to keep p99 latency.
Metadata scalability: centralized metadata managers (masters) simplify coordination but are single points; masterless (Dynamo-style) scales better but complicates coordination and repairs.
Conflict resolution: use vector clocks, last-write-wins, or CRDTs depending on tolerance for lost updates vs complexity of merges.
Multi-region design: active-passive reduces cross-region latencies; multi-active requires conflict resolution and either global consensus or per-region leader + cross-region replication.
Operational metrics: track p50/p95/p99 latency, write-throughput, compaction stall times, replication lag, and repair backlog to detect hotspots early.

Worked example — "Design a distributed key-value store"

First 30s: ask workload: read/write ratio, object sizes, expected QPS, latency SLOs (e.g., p99 < 50ms), data durability (RPO/RTO), single or multi-region. Skeleton answer pillars: (1) partitioning with consistent hashing and virtual nodes for balancing, (2) replication using Raft per partition for strong consistency (or Dynamo-style quorums if availability prioritized), (3) storage engine choice: LSM-tree for write-heavy or B-tree for read-heavy, (4) rebalancing and metadata via a small metadata service, (5) monitoring/operational concerns (compaction tuning, backpressure). Tradeoff to flag: choosing Raft yields linearizable reads/writes but adds an RTT on writes and a leader bottleneck — acceptable for strict correctness but costly at very high write QPS. Close with: if I had more time I'd detail compaction strategy, per-partition capacity planning, shard split heuristics, and a failure-injection plan to validate p99 SLAs.

A second angle — "Design an object storage system (S3-like) for large blobs"

Frame differences: objects are large (MB–GB), favor multipart upload, and storage economics push toward erasure coding rather than 3× replication. Key design shifts: chunking objects into blocks, storing block metadata separately, implementing multi-part commits to provide resumability, and background repair (reconstruction after drive/node loss). Consistency model can be weaker for object head/list operations (eventual) while PUT/GET semantics are linearizable per object if required. Operationally, optimize for throughput and streaming reads; caching and CDN integration matter more than per-object low-latency metadata reads.

Common pitfalls

Pitfall: Ignoring metadata scale.
Designs that shard only data but centralize metadata (object index, partition map) break at millions of keys; propose scalable metadata (partitioned or hierarchical) and quantify its expected size and lookup latency.

Pitfall: Choosing consensus without workload questions.
Picking Raft by default without knowing read-heavy or geo-replicated needs can cause needless latency; state your consistency-latency tradeoff and justify the chosen protocol.

Pitfall: Underestimating compaction and repair costs.
A tempting answer is “use LSM for writes” without accounting for compaction IO spikes and repair bandwidth during rebalancing; include throttles, backoff, and operational knobs.

Connections

Interviewers may pivot to distributed transactions (two-phase commit, MVCC), caching/CDN strategies for read amplification, or hardware-aware design (NVMe, SSD wear-leveling) to optimize latency and cost. Be prepared to discuss monitoring/SLIs and how design choices affect alerting and SLOs.