Interview conceptSystem Design

Reliable Distributed Systems And Idempotency

Asked of: Software Engineer

Last updated

Architecture showing idempotent API requests, transactional outbox publishing, at-least-once event delivery, inbox deduplication, and reconciliation of external effects.

What's being tested

This topic tests whether you can design reliable distributed workflows when requests, messages, and downstream effects may be retried, duplicated, delayed, or reordered. The interviewer is probing for a precise understanding of idempotency, at-least-once delivery, transactional boundaries, concurrency control, and recovery after partial failure—not a claim that “exactly once” magically exists. Coinbase cares because trading, balances, notifications, and custody-related operations must remain correct when services, networks, consumers, and external providers fail independently.

Core knowledge

  • Idempotency means repeating the same logical operation produces the same externally visible result. Require a client-supplied idempotency key, scope it to the authenticated account and operation, and return the originally persisted response for later retries.

  • An idempotency record should atomically reserve (tenant, operation, key), store request-hash and status, then persist the final result. Reject the same key with different parameters; otherwise concurrent requests can accidentally represent two operations.

  • At-least-once delivery is the practical default: a broker may redeliver after a consumer commits work but crashes before acknowledging. Consumers therefore need an inbox or deduplication table keyed by stable event_id, not merely a broker offset.

  • Exactly-once semantics usually mean exactly-once effects within a narrowly defined transactional boundary. They do not guarantee one external email, webhook, exchange order, or network request happened once; model external calls as retryable and reconcile outcomes.

  • Use a transactional outbox when a database mutation and event publication must agree: commit business state plus an outbox row in one transaction, then publish asynchronously. A relay may publish duplicates, so consumers remain idempotent.

  • For event consumers, an inbox pattern records event_id and business effects in one transaction. If the event is already recorded, acknowledge it without reapplying the effect; retain dedupe state at least as long as replay and retry windows.

  • Ordering is normally scoped, not global. Partition by account, order, or notification recipient; attach monotonically increasing sequence numbers, detect gaps, and decide whether to buffer, reject, or request replay when sequence=9 arrives before 8.

  • Backpressure requires bounded queues, per-consumer limits, leases, exponential backoff with jitter, and a dead-letter or retry queue. Isolate slow consumers so one blocked webhook or mobile client cannot exhaust shared worker capacity.

  • For financial state, use a double-entry ledger: every transaction has balanced debit and credit postings, with ∑credits=∑debits\sum credits=\sum debits. Make the ledger append-only, derive balances from postings or controlled snapshots, and use database constraints to prevent partial application.

  • Concurrency control prevents two valid requests from overspending or filling the same order incorrectly. Choose serializable transactions, optimistic version checks (UPDATE ... WHERE version = v), or per-key serialization; retries must preserve the original idempotency key.

  • Retry safety depends on classifying failures. A timeout after an external submission is ambiguous, while a validation error is definitive; retry ambiguous operations only with a provider idempotency key or reconciliation query, never by blindly submitting again.

  • Durability and recovery require explicit state machines such as PENDING → APPLIED → PUBLISHED → DELIVERED, durable timestamps, and replayable events. Recovery workers scan stale leases and reconcile states; “marked complete before doing the work” is unsafe.

  • Consistency choices should follow the invariant: strong consistency for available-balance checks and one-time state transitions, eventual consistency for notification fanout and search views. State the user-visible consequence of stale reads rather than calling everything “eventually consistent.”

  • Observability should expose idempotency-hit rate, duplicate-event rate, outbox age, consumer lag, retry counts, dead-letter volume, reconciliation mismatches, and p99 processing latency. Correlation IDs and event IDs must follow a request across services without leaking sensitive payloads.

Worked example: Deliver Limit-Order Fill Notifications under Consumer Backpressure

I would first clarify delivery channels, notification latency, retention, per-order ordering guarantees, duplicate tolerance, fanout scale, and whether a provider offers idempotent webhook submission. I would declare that fills are durable facts, delivery is at least once, and a notification may be delayed but must not be silently lost. I would organize the answer around an append-only fill event, a transactional outbox or durable event log, per-recipient fanout with scoped ordering, and isolated delivery workers with retry and dead-letter handling. Each fill gets a stable event_id and sequence number, while each consumer stores an inbox entry before acknowledging so redelivery cannot create a duplicate notification. To handle backpressure, I would use bounded per-consumer queues, leases, exponential backoff with jitter, and quotas so one slow recipient cannot block other recipients or the matching system. I would explicitly trade strict ordering for a recipient against latency and storage cost, preserving order per order or account rather than imposing a global sequence. For external providers, I would treat timeout-after-submit as ambiguous and use their idempotency key or a reconciliation endpoint before retrying. If I had more time, I would detail retention, replay tooling, provider-specific rate limits, and metrics such as outbox age and delivery lag.

A second angle

In Design a bank account service, the same principle applies to money movement rather than notification delivery. The key framing changes from consumer backpressure to atomic balance invariants, authorization, concurrent transfers, and auditability. A client retry of POST /transfer must return the original transfer result, while the debit, credit, idempotency record, and ledger entries commit atomically. The difficult failure is an uncertain response after commit, so querying by idempotency key or transfer ID is safer than issuing a second transfer. Reconciliation then compares derived balances, ledger postings, and external settlement records.

Common pitfalls

Pitfall: Saying “use exactly-once processing” as though it solves retries across services is analytically incomplete. Explain the transactional boundary, then combine at-least-once delivery, durable deduplication, idempotent state transitions, and reconciliation for external effects.

Pitfall: Jumping directly into Kafka partitions, databases, and worker pools without first stating invariants and failure assumptions makes the design difficult to evaluate. Start with what must never happen—double debit, lost fill, or out-of-order user-visible state—and map each mechanism to that invariant.

Pitfall: Treating a timeout as proof that an external operation failed is a depth mistake. A timeout means the outcome is unknown; use provider-side idempotency, status lookup, durable operation state, and a reconciliation loop rather than blindly retrying.

Connections

An interviewer may pivot to transaction isolation, leader election and leases, message ordering, transactional outbox/inbox, or reconciliation jobs. They may also ask how to design APIs and schemas so retries remain safe during deploys, failovers, and regional outages.

Further reading

Practice questions

Related concepts