Distributed Storage, Caching And Consistency

What's being tested

These designs test whether you can turn vague requirements into explicit consistency, durability, latency, and availability contracts, then choose storage and caching mechanisms that satisfy them. The interviewer is probing practical distributed systems reasoning: replication, partitioning, cache invalidation, ordering, retries, and failure recovery—not just naming databases. Salesforce operates multi-tenant enterprise systems where one tenant’s traffic, data, or outage must not compromise another’s correctness or isolation.

Core knowledge

• Consistency models define what reads may observe: linearizability gives a single real-time order, sequential consistency preserves each client’s order, and eventual consistency allows stale reads. State which operations require which guarantee instead of labeling the whole system “strongly consistent.”

• Replication improves availability and read throughput but creates divergence. Leader-based replication simplifies writes and ordering; multi-leader or leaderless designs improve locality but require conflict resolution, version vectors, last-write-wins, or application-specific merges.

• For quorum systems, with replication factor $N$, choose write quorum $W$ and read quorum $R$ such that $W+R>N$ to overlap replicas. This detects recent writes, but does not automatically provide linearizability under clock skew, sloppy quorums, or read-repair lag.

• Cache-aside keeps the database authoritative: read cache, fetch on miss, then populate; write the database first and invalidate or update the cache. Protect against stale repopulation by using versioned values, compare-and-set, or delayed invalidation strategies.

• TTL bounds staleness but does not prevent stale reads before expiry. For correctness-sensitive mappings, use explicit invalidation, generation numbers, or a strongly consistent metadata store; add jitter to TTLs to avoid synchronized expiration storms.

• Hot keys can overload one cache shard or database partition even when aggregate traffic is moderate. Replicate popular immutable values, request-coalesce concurrent misses, shard counters, or route reads across replicas; never assume horizontal scaling fixes skew automatically.

• Partitioning should isolate tenants and distribute load using tenant ID, conversation ID, or a composite key. Hash partitioning balances traffic; range partitioning supports scans but risks hot ranges. Large tenants may need dedicated shards or subpartitioning.

• Idempotency makes retries safe. Assign a client request or event ID, persist it with the result, and enforce a uniqueness constraint in Postgres or the chosen durable store; a timeout means “unknown outcome,” not automatic permission to duplicate work.

• Ordering must be scoped realistically. A per-conversation sequence number or partition key can provide ordered messages, while global ordering is expensive and often unnecessary. Persist sequence metadata and define behavior for gaps, duplicates, and late arrivals.

• Durability requires acknowledging writes only after the configured persistence boundary—such as replicated log commit or quorum acknowledgment. Separate durability from availability: accepting a write during a partition may preserve availability but weaken read-after-write guarantees.

• Failure handling should cover cache loss, replica lag, leader failure, network partitions, retries, and partial fan-out. Use bounded timeouts, exponential backoff with jitter, circuit breakers, dead-letter or repair workflows, and observability for p50, p99, error rate, lag, hit rate, and stale-read rate.

• Tenant isolation includes authorization at every read and write path, tenant-aware cache keys, quotas, encryption boundaries, and noisy-neighbor controls. A cache key such as objectId without tenant context can become a cross-tenant data leak even when the database query is correct.

Worked example: Design a Multi-Tenant Enterprise Messaging System

A strong candidate first clarifies tenant scale, peak messages per second, maximum acceptable message loss, required ordering scope, retention, search freshness, presence accuracy, and whether direct messages need stronger guarantees than presence. They should declare that durable message history is authoritative, real-time delivery is best-effort and replayable, and ordering is per conversation rather than global. The answer can then be organized around partitioned durable storage, an event-delivery path, read and search models, and authorization plus tenant isolation. Messages might be appended with a conversation sequence, published through Kafka-like partitions, persisted before acknowledgment, and delivered through WebSocket gateways with offline replay from the history store. Read positions and presence should be modeled separately because read positions need durable monotonic updates while presence tolerates staleness. A key tradeoff is whether to synchronously fan out to every participant or store once and fan out on read for large channels; the latter protects writes from celebrity-channel explosions but increases read work. Caches can accelerate conversation metadata and recent history, but tenant-scoped keys and explicit invalidation are required to avoid stale permissions or cross-tenant exposure. The candidate should close by saying they would next quantify partition hot spots, failure recovery, retention compaction, search indexing lag, and load-test reconnect storms.

A second angle

For Design a URL-Shortening Service, the central consistency question is different: a short code must never ambiguously map to two destinations, while redirects can usually tolerate briefly stale cache entries. Use a durable uniqueness constraint or collision-resistant ID allocation for creation, then cache immutable or versioned redirect mappings at the edge. A delete or expiration policy introduces a cache invalidation race, so include generation checks, bounded TTLs, and an explicit policy for redirects already cached by clients or CDNs. Multi-region creation may favor a globally unique ID scheme and asynchronous replication, whereas a single-writer metadata region makes collision and update semantics simpler.

Common pitfalls

Analytical mistake: Saying “use eventual consistency everywhere for scale” ignores operation-specific correctness. A better answer classifies writes, reads, permissions, counters, and presence separately, then states where linearizable or read-after-write behavior is required.

Pitfall: Treating a cache hit as proof that the value is current, authorized, or tenant-safe.

Communication mistake: Jumping directly to Cassandra, Redis, or Kafka without requirements makes the design sound cargo-culted. State assumptions, sketch request and failure flows, and explain why each component satisfies a stated latency, durability, or isolation requirement.

Depth mistake: Discussing replication only in the healthy path misses the interview’s most valuable signal. Explain what happens during leader loss, network partition, duplicate delivery, cache outage, replica lag, and a tenant generating disproportionate traffic.

Connections

The interviewer may pivot to CAP tradeoffs, failure detectors, event sourcing, CQRS, or multi-region disaster recovery. Be ready to connect consistency choices to schema design, idempotent APIs, authorization enforcement, and operational metrics without drifting into infrastructure ownership outside the service boundary.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s treatment of replication, partitioning, consistency, transactions, and stream processing.

• Dynamo: Amazon’s Highly Available Key-value Store — seminal discussion of quorum-style availability, conflict resolution, and eventual consistency.

• Caching in the Distributed World — Stripe’s idempotency-key pattern for making retried distributed requests safe.