System Design: Scale a Key–Value Store with Consistent Hashing

Context

You have a single-server key–value store that must scale horizontally across many servers while maintaining high availability and low latency. Assume point lookups/updates by key (no multi-key transactions), and tunable consistency is acceptable.

Tasks

1. Partition keys with consistent hashing, using virtual nodes (vnodes). Explain the hash ring, token assignment, and how clients/gateways route requests.
2. Describe node joins and leaves: how they trigger minimal data movement and how rebalancing works.
3. Define replication and quorum choices: pick N (replication factor), R (read quorum), W (write quorum), and justify trade-offs.
4. Specify read/write paths: coordinator behavior, conflict/version handling, and read repair.
5. Explain failure detection, re-replication, and recovery: membership, hinted handoff, anti-entropy, and rebuild.
6. Address hot keys, skew, and load balancing.
7. Ensure data migration safety during ownership changes; describe monitoring and rolling upgrades.
8. Compare alternatives (e.g., directory service, range partitioning) and justify trade-offs.

Deliver a diagram-free written design that covers the above with clear assumptions, pitfalls, and metrics/SLOs.