Transactional Consistency, Idempotency, And Exact Contracts

What's being tested

These designs test whether you can turn ambiguous business behavior into exact contracts: state transitions, invariants, API semantics, numeric representations, timestamp rules, and error behavior. You must demonstrate transactional consistency under concurrency, idempotency under retries, and deterministic outcomes that can be explained after a failure. Citadel cares because financial and high-value transactional systems cannot silently oversell, double-submit, lose a fill, or produce a reconciliation result that changes depending on execution order.

Core knowledge

• Invariants are non-negotiable truths maintained across every transition: for inventory, `on_hand` >= reserved >= 0 and `available` = on_hand - `reserved`; for enrollment, `enrolled_count` <= capacity ``; for reconciliation, every break has a traceable source and policy decision.

• Use a relational database such as Postgres when correctness depends on atomic updates, uniqueness, and constraints. A transaction should group all-or-nothing changes, while `SELECT ... FOR UPDATE`, serializable isolation, or optimistic version checks protect contested rows.

• An idempotency key identifies one logical client operation, not one network attempt. Store (client_id, key, request_fingerprint, status, response) with a unique constraint; replaying the same request returns the original result, while a changed payload is rejected.

• Idempotency does not mean “the request can never fail.” Persisting a durable operation record before an external side effect, then using an outbox pattern, lets workers retry publication without losing the committed database event.

• Define linearization points explicitly: the instant an enrollment consumes a seat, an order reserves stock, or a reconciliation result becomes authoritative. Every concurrent execution should appear equivalent to some serial ordering around that point.

• Exact monetary behavior requires integer minor units or a decimal type with declared scale and rounding mode; never use binary floating point for dollars. State whether totals use half-even, half-up, or another policy, and apply it consistently.

• Timestamp contracts need a timezone, precision, and boundary convention. Prefer UTC instants with half-open intervals, $[start, end)$, so adjacent windows do not double-count events; define an allowed clock-skew or matching tolerance separately from ordering.

• Optimistic concurrency control stores a version or compare-and-swap field: update only when version = expected_version, then increment it. It scales better than long locks when conflicts are rare, but callers must handle retries and distinguish conflict from absence.

• Multi-item operations require atomicity across all items: validate availability, lock or conditionally update rows in a stable key order, write the reservation, and commit once. Stable ordering reduces deadlocks; partial success should be an explicit contract, never an accidental outcome.

• Determinism requires stable tie-breakers and centralized policy ownership. A top-ten query might order by (score DESC, trade_id ASC); reconciliation should assign one owner for tolerance, partial-fill, and rounding rules instead of duplicating slightly different logic across callers.

• Read-after-write semantics must be named. A successful write can return committed state from the primary, while an eventually consistent replica may lag; avoid claiming “confirmed” when the response only means an asynchronous command was accepted.

• Recovery paths are part of the contract: timeouts can mean “unknown outcome,” not failure. Expose a status lookup by operation ID, retain request and response history, and make expiration, cancellation, duplicate submission, and late completion transitions explicit.

Worked example

For Design Inventory Availability and Order Placement, I would first clarify whether reservations are hard commitments, how long they live, whether orders may contain multiple items, and what a client should see after a timeout. I would declare a relational source of truth, integer quantities, a reservation TTL, and the invariant available = on_hand - reserved >= 0. I would organize the answer around the data model and invariants, the atomic placement transaction, retry-safe API semantics, and expiration/reconciliation workers. For placement, I would validate the request, acquire item rows in sorted SKU order, conditionally reserve every item, insert an idempotency record, and commit the order and reservations together. The API would require an idempotency key and request fingerprint, returning the original order for a duplicate identical request and a conflict for a reused key with different contents. I would explicitly compare pessimistic row locks with optimistic version checks: locks provide a straightforward correctness story for hot inventory, while optimistic updates may improve throughput when conflicts are uncommon. I would explain that a client timeout leaves the result unknown, so GET /orders/{id} or an idempotency replay must resolve it rather than blindly submitting again. If I had more time, I would add crash recovery, reservation expiration races, metrics for contention and oversell attempts, and a load test for hot SKUs.

A second angle

For Define a Reliable Trade-Reconciliation Library Contract, the same principles apply, but the central risk is not overselling; it is producing an ambiguous or non-reproducible decision from imperfect records. The contract should define exact amount types, timestamp tolerance, partial-fill treatment, canonical trade identity, and a structured break containing both inputs and the applied policy. Idempotency appears as deterministic evaluation: rerunning the same normalized records and policy version must produce the same result. The key design choice is to keep policy ownership in one library boundary, rather than letting each caller independently decide rounding or tolerance behavior.

Common pitfalls

Pitfall: Treating a database transaction as a complete distributed transaction. A local commit cannot atomically commit a remote payment, message broker, or downstream order; use an outbox, durable status, and compensating state transitions instead.

Pitfall: Saying “use an idempotency key” without specifying storage, scope, payload mismatch behavior, retention, or the timeout case. A stronger answer defines the uniqueness constraint, response replay, fingerprint validation, and how a client discovers an unknown result.

Pitfall: Going deep on infrastructure while leaving the contract vague. Naming `Kafka` partitions or caches does not answer whether two concurrent requests can both reserve the last unit; state the invariant, linearization point, isolation mechanism, and observable error semantics first.

Connections

An interviewer may pivot to isolation levels, deadlock handling, and serializability; outbox/inbox messaging and at-least-once delivery; or state-machine design for cancellation and expiration. They may also ask about schema migrations, auditability, and testing with fault injection and concurrent property-based tests.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s treatment of transactions, consistency, replication, and distributed-system failure modes.

• Stripe API Idempotent Requests — a concrete production contract for retry-safe mutation APIs.

• Transaction Processing: Concepts and Techniques — Jim Gray and Andreas Reuter; a rigorous reference for atomicity, recovery, locking, and transaction correctness.