Design a distributed key-value store

Read the full interview experience this question came from →

Quick Overview

Design a distributed key-value store evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design a distributed key-value store

Company: LinkedIn

Role: Machine Learning Engineer

Category: System Design

Difficulty: hard

Interview Round: Technical Screen

Design a distributed key–value storage service. Requirements: high availability across availability zones, horizontal scalability to billions of keys, low-latency reads/writes, per-key read-after-write consistency, durability, TTL support, conditional updates (CAS), and optional range scans. Define the API and data model. Choose a sharding strategy (consistent hashing vs range), replication model (leader–follower vs leaderless), and lay out read/write paths, quorum choices, and conflict resolution. Select on-disk structures (e.g., LSM vs B+Tree), compaction, indexing, and hot-key mitigation. Explain rebalancing, failure detection, recovery, backups, and disaster recovery. Include observability (metrics, tracing, alerts) and discuss CAP trade-offs and testing methodology.

Overview: Design a distributed key-value store evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Read the full LinkedIn Machine Learning Engineer interview experience this question came from

|Home/LinkedIn/System Design
LinkedIn logo
LinkedIn
Jul 16, 2025
hardMachine Learning EngineerTechnical ScreenSystem Design
13
0

Design a distributed key-value store

Design a Distributed Key–Value Store (Technical Screen)

Context

You're designing a cloud-native, multi-tenant key–value (KV) storage service for internal ML/analytics platforms. The service must support billions of keys with low-latency reads/writes and be highly available across availability zones (AZs). Some workloads require conditional updates (CAS) and per-key read-after-write consistency; others occasionally need range scans (e.g., prefix scans for model features).

Functional Requirements

  • API supports: Get, Put/Upsert, Delete, Batch, Compare-And-Set (CAS), TTL per key, and optional range scans.
  • Data model: binary values; metadata includes TTL, version, and optional attributes.
  • Per-key read-after-write consistency.
  • Optional range scans (prefix or ordered key ranges).

Non-Functional Requirements

  • High availability across AZs.
  • Horizontal scalability to billions of keys.
  • Low latencies: reads/writes p99 under tight SLOs (assume single-digit ms within an AZ when cache hits; low tens of ms on stable storage access).
  • Durability.
  • Hot-key mitigation.
  • Observability (metrics, tracing, alerts).

Design Tasks

  1. Define the API and data model, including error semantics and consistency options.
  2. Choose sharding strategy: consistent hashing vs. range partitioning; justify how to support range scans.
  3. Choose replication model: leader–follower vs. leaderless. Define read/write paths, quorum choices, and conflict resolution.
  4. Select on-disk structures (e.g., LSM vs. B+Tree), compaction strategy, indexing, TTL handling, and caching.
  5. Explain hot-key mitigation strategies.
  6. Explain rebalancing, failure detection, recovery, backups, and disaster recovery.
  7. Provide observability plan (metrics, tracing, alerting).
  8. Discuss CAP trade-offs and tunable consistency.
  9. Outline testing methodology for correctness, performance, and resilience.

Clarifying Questions to Ask Guidance

  • Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
  • State explicit assumptions before making sizing or architecture decisions.
  • Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

  • A scoped requirements summary with concrete non-goals and success metrics.
  • API, data model, architecture, consistency, capacity, and operations.
  • Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
  • A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

  • What breaks first at 10x traffic or data volume?
  • How would you degrade gracefully during dependency failures?
  • What metrics and alerts would prove the design is healthy after launch?

Browse More Questions

More System Design•More LinkedIn•More Machine Learning Engineer•LinkedIn Machine Learning Engineer•LinkedIn System Design•Machine Learning Engineer System Design