Distributed Systems Design At Scale

What's being tested

These designs test whether you can decompose a distributed system into clear APIs, state ownership, data paths, and failure boundaries rather than jumping directly to technologies. Across configuration delivery, caching, input reconstruction, log analytics, and ad serving, the interviewer probes partitioning, ordering, consistency, durability, latency, and recovery under realistic failure modes. Amazon cares because Software Engineers must build services that remain correct and operable when machines fail, traffic spikes, messages are duplicated, and requirements conflict. A strong answer makes guarantees explicit, quantifies assumptions, and explains tradeoffs instead of claiming “exactly once” or “strong consistency” without defining scope.

Core knowledge

• Requirement clarification should establish read/write QPS, object sizes, retention, latency targets, availability, durability, geographic scope, and acceptable staleness. Convert vague goals into measurable SLOs such as p99 < 100 ms, 99.99% availability, or recovery point objective (RPO) of zero.

• API contracts should define resource identity, version fields, pagination, timeouts, retries, authentication, and error semantics. Use an idempotency key for retried writes, and return a monotonic version or ETag so clients can perform conditional updates with If-Match.

• Partitioning distributes keys across nodes using consistent hashing or rendezvous hashing. Virtual nodes reduce imbalance, but rebalancing still requires bounded data movement, throttling, and a clear behavior for requests during ownership transitions.

• Replication improves availability but introduces consistency choices. Leader-based replication simplifies ordered writes; quorum replication uses $R + W > N$ for read/write intersection, while $R + W \leq N$ permits stale reads. State exactly which operations require a leader or quorum.

• Consistency is per operation and per data type, not a single system-wide label. Configuration versions and reconstructed input may require ordered, linearizable updates; cache reads may tolerate bounded staleness; log aggregation can usually be eventually consistent with correction for late events.

• Ordering needs a defined scope: per key, per client, per partition, or globally. A sequence number plus source identifier supports deduplication and gap detection; a global total order is expensive and often unnecessary. Clock timestamps alone do not establish causal order because clocks skew.

• Durability and recovery typically combine a replicated write-ahead log, durable snapshots, and replay. Define the acknowledgment point: acknowledging before replication risks data loss, while waiting for multiple replicas increases write latency. Recovery must address partial writes, duplicate replay, and corrupted or missing segments.

• Invalidation and cache correctness require a write policy: cache-aside, write-through, or write-behind. Cache-aside is simple but permits stale values and stampedes; use versioned values, delete-after-write, leases, request coalescing, and randomized TTL jitter to control races and synchronized expiry.

• Messaging semantics are normally at-least-once, so consumers must be idempotent. Durable queues or logs such as Kafka support replay and backpressure; dead-letter queues isolate poison messages. “Exactly once” generally means an end-to-end effect achieved with transactions, deduplication, or idempotent state transitions.

• Streaming and aggregation should separate ingestion from query serving. Partition logs by tenant or time, maintain indexes for exact attributes, and use time buckets for windows and histograms. Late events require a watermark or correction strategy; high-cardinality substring search may need inverted indexes rather than full scans.

• Hotspots and overload are first-class design concerns. A popular configuration key, cache key, advertiser, or log tenant can overload one partition; mitigate with replication, request hedging only when safe, admission control, batching, load shedding, and exponential backoff with jitter.

• Observability and operability should expose request rate, error rate, saturation, p50/p95/p99 latency, replication lag, cache hit ratio, queue depth, dropped events, and version gaps. Include health checks, canaries, alerts tied to SLOs, tracing correlation IDs, and a rollback or fail-forward procedure.

Worked example

For Design a configuration subscription service, I would first ask whether updates must be globally ordered, how many clients and subscriptions exist, the maximum propagation delay, and whether a disconnected client must receive every missed version. I would state an assumption such as per-configuration-key ordering, durable storage, at-least-once notifications, and eventual convergence within a defined SLO. I would organize the answer around the write API and version store, subscription/connection management, durable event distribution, and client catch-up plus observability. A writer would persist the new value and monotonically increasing version before publishing an event through a replicated log, while subscribers would acknowledge versions and fetch missing ranges after reconnecting. I would make delivery idempotent by including (key, version) and requiring clients to ignore versions at or below their applied version. The key tradeoff is whether to maintain long-lived connections for low latency or use polling for simpler scaling and firewall behavior. I would explicitly avoid promising global ordering if per-key ordering satisfies the requirement. If I had more time, I would detail multi-region conflict handling, authorization propagation, snapshot compaction, and a rolling schema-migration plan.

A second angle

For Design a Distributed In-Memory Cache: Partitioning, Failover and Invalidation, the dominant framing changes from durable event delivery to low-latency reads under bounded inconsistency. Start by separating cache ownership, replication, eviction, and database interaction, then define behavior when a node fails or a cached value races with a write. Consistent hashing and replica promotion address availability, while versioned cache entries and write-through or explicit invalidation address correctness. The interviewer may specifically test stampede prevention, hot-key replication, and whether failover can return stale data. A strong answer states that the database remains the source of truth and explains how recovery repopulates the cache without overwhelming it.

Common pitfalls

Analytical mistake: Saying “use a distributed cache” or “use Kafka” without defining the consistency and delivery guarantee is incomplete. State the invariant first—for example, “a client never applies version 7 after version 8”—then select mechanisms that enforce it.

Pitfall: Treating timestamps as a total order, or claiming quorum reads are always strongly consistent, ignores clock skew, leader changes, stale replicas, and the actual acknowledgment protocol.

Communication mistake: Jumping into node counts and schemas before clarifying scale makes a technically reasonable design appear unfocused. Lead with assumptions, draw the request and failure paths, and periodically tie each component back to a stated requirement.

Pitfall: Listing every possible feature—multi-region active-active, global ordering, exactly-once delivery—without prioritizing the baseline design signals poor scope control.

Depth mistake: Designing only the happy path misses what these systems are intended to test. Walk through duplicate messages, lost subscribers, partition movement, replica failure, cache/database races, backpressure, and recovery from durable state.

Pitfall: Saying “retry on failure” without timeouts, exponential backoff, jitter, idempotency, and a retry budget can turn a partial outage into a cascading failure.

Connections

The interviewer may pivot to CAP theorem, leader election, consistent hashing, backpressure, idempotent APIs, or multi-region replication. They may also ask for a focused implementation of an LRU cache, a bounded queue, a rate limiter, or a versioned subscription client.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s detailed treatment of replication, partitioning, consistency, logs, and stream processing.

• Amazon Dynamo — seminal discussion of quorum-style replication, consistent hashing, and availability-oriented design.

• The Tail at Scale — why p99 latency, hedging, and fanout behavior matter in distributed services.