Distributed Systems Interview Questions for Senior Engineers: Consistency, Replication, and Failure

A mid-level candidate can explain replication. A senior candidate can tell you exactly which write may be lost, which read may be stale, what a timeout means, and how the system returns to a trustworthy state after part of it fails.

That difference is why distributed systems interviews become harder with seniority. The interviewer is not looking for the largest architecture or the most named algorithms. They are testing whether you can define a client-visible contract, make a defensible trade-off, and reason through partial failure without hand-waving.

Use PracHub interview questions with written solutions to practice these decisions under realistic prompts. Attempt each question before opening the solution, then force yourself to trace one write, one read, and one failure from end to end.

Quick answer: what senior distributed systems interviews test

A strong senior answer starts with the guarantee the product needs, not a preferred database. It then explains topology, acknowledgement, read eligibility, partition behavior, retry semantics, recovery, and the evidence that proves the system is healthy.

The core habit is to make every success response mean something precise. If the service returns 200 after a write, has the leader only stored it in memory? Is it durable on one disk, a majority of replicas, or another region? Can a later read miss it? A senior candidate answers those questions before claiming the design is reliable.

Use the senior answer trace

The most reliable way to answer is to move through five checkpoints: contract, topology, commit, failure, and recovery. This prevents a diagram from becoming a collection of boxes with undefined behavior.

1. Contract: what must the client observe?

Start with business invariants. A social feed may tolerate delayed propagation; a payment authorization cannot tolerate a duplicated charge; inventory cannot confirm the same scarce unit twice. Translate the product rule into read-after-write, monotonic reads, causal ordering, uniqueness, or transactional isolation.

2. Topology and commit: where is the authority?

Show partitioning, replica placement, the write authority, and the acknowledgement boundary. Then trace the write from request to durable log, replica acknowledgement, response, and later read. If multiple regions accept writes, explain conflict detection and resolution instead of assuming convergence.

3. Failure and recovery: what happens when the happy path breaks?

Inject one failure at a time: leader crash, slow replica, lost response, network partition, region outage, or replay after restart. State what clients see, what state may diverge, how authority is fenced, and how replicas reconcile. End with the metric or test that confirms recovery.

Consistency interview questions

What does strong consistency mean in this design?

This is often a trap because "strong" is underspecified. Ask whether the interviewer means linearizable operations, serializable transactions, read-your-writes, or simply no stale replica reads. Linearizability respects real-time ordering for operations on an object; serializability makes concurrent transactions equivalent to some serial order, but does not by itself require that order to match wall-clock time.

A strong answer chooses the smallest guarantee that preserves the invariant. PostgreSQL's documentation is useful here: its serializable isolation may abort a transaction when a concurrent execution cannot be reconciled with a serial order. The application still needs a retry policy for that explicit failure.

When is eventual consistency acceptable?

Eventual consistency is acceptable when temporary divergence does not violate a critical invariant and the product has a clear convergence rule. Examples include like counts, derived search indexes, and some preference propagation. State the staleness bound users can tolerate, how conflicts resolve, and whether a session must read its own writes.

Azure Cosmos DB exposes strong, bounded-staleness, session, consistent-prefix, and eventual models. The interview lesson is not to memorize those names. It is to show that consistency can be selected per access pattern and that stronger cross-region guarantees affect latency and availability.

How would you prevent a user from seeing time move backward?

Discuss session guarantees such as monotonic reads and read-your-writes. Options include routing a session to a sufficiently fresh replica, carrying a version or session token, or reading from the leader after a write. A senior answer also explains what happens when the preferred replica is unavailable: wait, fail, or relax the guarantee with an explicit product decision.

Replication interview questions

Leader-follower or leaderless replication?

Leader-follower replication provides one write order and makes many invariants easier to reason about, but leader availability and failover become critical. Leaderless designs can accept operations at multiple replicas and improve write availability, but must resolve concurrent versions and reason carefully about quorums, repair, and client coordination.

Do not choose from ideology. Start with write contention, latency geography, consistency needs, failure domains, and operational maturity. For a deeper treatment of election, lag, quorum reads, and split brain, use PracHub's database replication interview questions guide.

Does R + W > N guarantee a fresh read?

Quorum intersection is useful, but the equation alone is not a complete correctness proof. Ask whether the system uses the same replica set, whether writes are versioned, how concurrent writes are resolved, whether sloppy quorums or hinted handoff are allowed, and whether the read waits for and reconciles enough responses.

A good answer traces a concrete case with one failed node and one delayed write. It identifies which version wins, who repairs stale replicas, and what the client sees when the operation times out after some replicas accepted it.

What does synchronous replication actually protect?

Synchronous replication protects the acknowledgement boundary you configure. A majority acknowledgement can survive some node failures, but it does not replace backups, prevent application corruption, or guarantee zero loss if replicas share a failure domain. Multi-region synchronous replication also puts wide-area latency on the write path.

State replica placement by zone and region, the tolerated failure count, and whether RPO is truly zero for acknowledged writes. Then explain how degraded quorum affects availability and how the system returns to the desired replication factor.

Failure and retry interview questions

The write timed out. Did it happen?

A timeout tells the caller that it did not receive a result before its deadline. It does not prove the server did nothing. The write may have committed and the response may have been lost, so blindly retrying can duplicate a charge, reservation, or job.

Use an idempotency key tied to the logical operation, persist the result, and return the same outcome on retry. Define key scope, retention, payload mismatch behavior, and concurrency handling. AWS's guidance on idempotent APIs emphasizes this exact problem: retries are safe only when repeated requests cannot create unintended extra effects.

How should a service behave during a network partition?

Name the partitioned components and decide per operation. A minority replica group may continue stale reads, reject writes, or serve a degraded feature. A majority group may elect a leader and continue, but old leaders must be fenced so they cannot accept writes after rejoining.

Avoid saying "CAP means choose consistency or availability" without defining availability for the requested operation. Eric Brewer's discussion of Spanner and CAP is a useful reminder that partitions are intervals with system-specific behavior, not a permanent database label.

How can retries turn a partial outage into a total outage?

When latency rises, callers time out and retry, multiplying work on an already constrained dependency. Add bounded retry budgets, exponential backoff with jitter, deadlines, concurrency limits, load shedding, and circuit breaking. Retry at one layer whenever possible so nested services do not create multiplicative attempts.

Google SRE documents how reduced capacity can redirect load to surviving replicas and create cascading failure. The senior signal is connecting client behavior, queues, resource saturation, and recovery rather than treating each service in isolation.

Consensus, ordering, and clocks

When do you need consensus?

Consensus is useful when nodes must agree on a sequence of state transitions, such as leader election, configuration changes, or a replicated log. Raft separates leader election, log replication, and safety, using majority agreement so a cluster can continue after a bounded number of failures.

Consensus does not automatically solve every application-level transaction, duplicate request, or cross-service workflow. Explain which small piece of state requires agreement and keep the consensus group scoped enough to operate predictably.

Can timestamps order distributed events?

Ordinary wall clocks drift and can move. Use monotonic clocks for elapsed time, logical or hybrid clocks for causal/version ordering, and explicit tie-breaking for concurrent writes. If you mention Spanner, explain that TrueTime exposes a bounded uncertainty interval and is specialized infrastructure, not evidence that any synchronized clock makes last-write-wins safe.

A worked senior scenario

Imagine a multi-region inventory service for limited-release products. The business invariant is that confirmed orders must not exceed available stock. Browsing can be slightly stale, but reservation and decrement must be authoritative.

A defensible design partitions stock by product, assigns one write leader per partition, and replicates its log across three zones. The service acknowledges a reservation only after a majority commits it. Requests carry idempotency keys, and the reservation record has an expiry so abandoned checkouts release inventory safely.

During a regional partition, the isolated minority cannot confirm reservations. It may continue serving clearly labeled availability estimates. If the leader fails, a majority elects a replacement; fencing prevents the old leader from writing when it returns. Recovery verifies committed log position, replica lag, reservation totals, and duplicate-key conflicts before traffic fully resumes.

That answer is not universally optimal. It is strong because every guarantee, acknowledgement, degraded behavior, and recovery step connects to the product invariant.

Practice with PracHub questions

Use these prompts to practice the full answer trace. Every complete title in the first column opens the question and written solution.

A seven-day preparation plan

Frequently asked questions

Do I need to memorize CAP theorem?

Know the model, but spend more time applying it. Name the actual partition, the operation, the guarantee, and the behavior of each side. Interviewers gain little from "CP versus AP" without a client-visible consequence.

Should I implement Raft or Paxos for interview preparation?

Most senior software interviews require conceptual accuracy, not a complete implementation. Be able to explain leader election, majority commit, log safety, membership, and failure. Implementing a small replicated log can deepen understanding, but it should not replace scenario practice.

How much math should I show for quorums?

Use enough math to prove intersection and tolerated failures, then discuss the assumptions behind it. Replica-set changes, concurrent versions, lag, sloppy quorums, and repair behavior often matter more than repeating R + W > N.

What is the best way to sound senior?

Do not add more technology. State the invariant, make the acknowledgement boundary explicit, distinguish ambiguous from definite failure, and close with recovery evidence. Precision sounds senior because it makes the design testable.

Final takeaway

Distributed systems interview questions for senior engineers are questions about meaning under failure. What does success mean? Which copy is authoritative? What may be stale? Can a retry duplicate work? Who may continue during a partition? How does the system prove that recovery is complete?

Practice those questions as one connected trace. PracHub gives you system design prompts with written solutions, but the real improvement comes from answering first, injecting failure, and defending each guarantee before comparing your approach.

Sources and further reading

• Microsoft Azure Cosmos DB global distribution and consistency models
• PostgreSQL transaction isolation documentation
• In Search of an Understandable Consensus Algorithm
• Amazon Builders' Library: Making retries safe with idempotent APIs
• AWS Architecture Blog: Exponential backoff and jitter
• Google SRE: Addressing cascading failures
• Google Research: Spanner, TrueTime and the CAP Theorem
• Google Cloud Spanner: TrueTime and external consistency

Research note: This guide was reviewed on August 22, 2026. Product implementations evolve, but the interview frameworks here focus on durable distributed-systems reasoning.