Concurrency Control

What's being tested

Interviewers are probing whether you can design financially correct backend systems under concurrent requests, retries, delayed execution, and partial failures. For Coinbase, this matters because account balances, transfers, scheduled payments, cashbacks, and ledger entries must preserve invariants even when many clients, workers, and services act at the same time. A strong Software Engineer answer should distinguish business correctness from database mechanics: locks, transactions, idempotency, and audit logs are tools for enforcing domain invariants like “money is neither created nor lost.” Expect the interviewer to test whether you can choose an appropriate consistency model, explain tradeoffs, and avoid hand-wavy “just use a transaction” answers.

Core knowledge

• Concurrency control is about preserving invariants when operations overlap. In banking-style systems, core invariants include non-negative available balance, exactly one effect per client request, immutable audit history, and double-entry balance preservation: total debits must equal total credits.

• ACID transactions are the default starting point for account mutations. Use a database like `Postgres` with BEGIN, row updates, constraints, and COMMIT to make multi-step operations atomic. The key interview move is explaining what rows are protected and at what isolation level.

• Pessimistic locking prevents conflicts by locking records before mutation, commonly with SELECT ... FOR UPDATE in `Postgres`. For a transfer, lock both account rows in deterministic order, such as ascending account_id, to avoid deadlocks when two opposite-direction transfers run concurrently.

• Optimistic concurrency control allows concurrent reads and detects conflicts at write time using a version column or compare-and-swap condition: UPDATE accounts SET balance = balance - 100, version = version + 1 WHERE id = ? AND version = ?. If zero rows update, retry with backoff.

• Isolation levels matter. READ COMMITTED can be acceptable with row locks and conditional updates, but can allow anomalies in more complex predicates. SERIALIZABLE gives the simplest correctness story but may reduce throughput due to transaction aborts; design retries for serialization failures.

• Idempotency keys protect against client retries, timeouts, and duplicate job execution. Store (client_id, idempotency_key) -> request_hash, status, response with a unique constraint. If the same key arrives with a different payload, return a conflict rather than executing a second mutation.

• Double-entry ledger design is stronger than updating balances directly. Represent each financial event as immutable ledger entries: debit one account, credit another, with a shared transaction_id. Balances become derived state, cached projections, or materialized summaries. This improves auditability and reconciliation.

• Available balance vs ledger balance is an important distinction. A card authorization, pending withdrawal, or scheduled payment may place a hold before final settlement. Avoid treating every pending operation as final money movement; model states like PENDING, POSTED, FAILED, and REVERSED.

• Atomic transfer usually requires checking funds and writing both sides in one transaction. The invariant is: from.available_balance >= amount before debit, and after commit, from.balance -= amount, to.balance += amount, plus immutable audit entries. Never debit in one transaction and credit later without a compensating design.

• At-least-once execution is common for schedulers and queues. A scheduled payment worker may run twice after a crash or visibility timeout. Make the payment execution idempotent with a stable payment_id or execution_id, not by assuming the scheduler fires exactly once.

• Deadlocks and hot accounts are realistic edge cases. Deterministic lock ordering reduces deadlocks; short transactions reduce lock hold time. For very hot entities, consider single-writer partitioning, per-account command queues, or ledger append with asynchronous projection rather than repeatedly updating one balance row.

• Auditability requires immutable records, not just logs. Application logs in `Datadog` or `CloudWatch` are not the source of truth. Store durable transaction records with who, what, when, amount, currency, idempotency key, previous state, resulting state, and links to external rail references.

Worked example

For “Design a bank account ledger,” a strong candidate should first clarify scope: “Are we supporting internal transfers only, or external payment rails too? Do balances need to be strongly consistent on reads? What throughput and latency should I assume? Are multi-currency accounts in scope?” Then declare a practical assumption: use a relational database such as `Postgres` for the core ledger because correctness and transactional constraints matter more than extreme write scale at first.

The answer skeleton should have four pillars. First, define the data model: accounts, ledger_transactions, ledger_entries, and optional balance_snapshots or materialized account balances. Second, define the write path: validate request, enforce idempotency, open a transaction, append balanced debit/credit entries, update balance projection, commit. Third, define concurrency control: lock affected accounts in deterministic order or use conditional versioned updates; retry on deadlocks and serialization failures. Fourth, define operational correctness: immutable audit trail, reconciliation jobs, monitoring for invariant violations, and recovery from partial external failures.

A specific tradeoff to flag is whether balances are computed from the ledger on every read or stored as a projection. Computing from the ledger is simplest and maximally auditable but becomes expensive as entries grow; maintaining a current_balance projection is faster but must be updated transactionally with ledger entries. A strong close would be: “If I had more time, I’d go deeper on external settlement states, multi-currency rounding, and backfill/reconciliation strategy, but the core design keeps every balance-changing event atomic, idempotent, and auditable.”

A second angle

For “Design a scheduled payments service,” the same concurrency ideas appear through delayed execution and retries rather than immediate user transfers. The scheduler should durably store payment intents with run_at, state, and an idempotency key, then workers claim due rows using a safe pattern such as SELECT ... FOR UPDATE SKIP LOCKED or a queue with visibility timeouts. The critical point is that claiming a job is not the same as executing the payment; the actual debit/credit operation must still be idempotent and transactionally protected. Here the design emphasis shifts from row-level account contention to job orchestration, duplicate execution, and state transitions like SCHEDULED -> PROCESSING -> SUCCEEDED or FAILED.

Common pitfalls

Pitfall: Saying “use a distributed lock” as the main correctness mechanism.

A lock service like `Redis` or `ZooKeeper` may help coordinate workers, but it should not be the source of financial truth. A better answer grounds correctness in database transactions, unique constraints, immutable ledger rows, and idempotent mutation APIs.

Pitfall: Treating idempotency as equivalent to concurrency control.

Idempotency prevents the same logical request from being applied twice; it does not prevent two different valid requests from overdrawing the same account. You still need row locks, conditional updates, serializable transactions, or another conflict-resolution mechanism for concurrent independent operations.

Pitfall: Over-indexing on high-level architecture and skipping invariants.

A tempting answer is to draw services, queues, and caches without saying exactly how money movement remains balanced. Interviewers will respond better if you state invariants early, then map each component to an enforcement mechanism: unique constraint for duplicate requests, transaction for atomicity, ledger entries for auditability, and reconciliation for external discrepancies.

Connections

Interviewers may pivot from this topic into idempotent API design, double-entry accounting, distributed transactions, event-driven architecture, or database isolation levels. They may also ask how your design changes when introducing `Kafka`, external payment processors, cached balances, or cross-region availability requirements.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s chapters on transactions, isolation, replication, and distributed systems are directly applicable to financial backend design.
• Stripe API Idempotent Requests — practical reference for the idempotency-key pattern used in payment-style APIs.
• PostgreSQL Transaction Isolation — concrete behavior of READ COMMITTED, REPEATABLE READ, and SERIALIZABLE in a real relational database.