Design a Distributed Key-Value Store for a Small Company's Internal Use

Design a distributed key-value store for internal use at a small company. Internal services store, read and delete values by key, and the data and traffic are spread over several machines. At the senior infrastructure level, the bar is less about drawing one architecture than about comparing the architectural options against the requirements and then proactively going deep on the distributed-systems problems the chosen design must handle. ```hint Compare before committing Sketch at least two ways to decide which machine owns a key and who routes a request to it, and note what each one costs when a machine joins, fails or receives a disproportionate share of the traffic. ``` ```hint Small scale still fails A modest data size does not remove machine failures, rebalancing or concurrent writes to the same key. Decide which of them the store must handle automatically, and what clients observe while it does. ``` ### Constraints and Clarifications - The setting is internal use at a small company, which suggests modest scale and a small team operating the system; weigh operational complexity alongside scale, latency and consistency. - Not given: data size, value sizes, request rates, latency and availability targets, consistency requirements, durability expectations, or the hardware available. Ask, or state your assumptions. ### Clarifying Questions - How much data is stored, how large are values, and what read and write rates are expected now and in a few years? - What consistency do clients need: must a read always see the latest acknowledged write, or is eventual consistency acceptable for some data? - What latency and availability targets matter, and is it acceptable to reject writes while part of the cluster is unreachable? - Are there access patterns beyond get and put, such as hot keys, range scans, expiry times or atomic compare-and-set? - How many engineers will operate the store, and which existing infrastructure (a coordination service, deployment tooling, object storage) can it rely on? ### What a Strong Answer Covers - Requirements and rough estimates before the design, and a design sized for a small company rather than a global service - A comparison of a central coordinator, a decentralized consistent-hashing design and a hybrid with a separate control plane, including the bottleneck and single-point-of-failure risk of each, before choosing - Partitioning and rebalancing that move little data when machines are added or removed, and how reads and writes are served while data moves - Replica placement, failure detection, request rerouting and recovery, with no single point of failure - A defined consistency model and a concrete answer for concurrent writes to one key - Hot keys and traffic skew, plus the storage engine, durability, observability and day-to-day operations ### Follow-up Questions - One key receives most of the traffic. What do you do if that traffic is reads, and what if it is writes? - You add a machine while clients keep writing. Walk through how data moves and what a client sees during the move. - Two clients write the same key at the same moment. What does a later read return, and how does your design guarantee it? - The component that holds the cluster metadata goes down. Which operations keep working, and which wait?