Stateful In-Memory Domain APIs

What's being tested

Candidates must model stateful in-memory domain APIs: correct domain objects, invariants (balances, ownership), and safe state transitions under concurrency. Interviewers probe API surface design, data structures, concurrency control, idempotency, and trade-offs between latency, correctness, and durability. Demonstrate clear assumptions, failure-mode handling, and incremental rollout plans a backend engineer would own.

Core knowledge

• Domain model: represent accounts, transactions, and merges as first-class objects with immutable transaction records and a mutable account state that references the latest valid balance and version number.

• Invariants: enforce atomic balance updates (no double-spend), non-negative balances if required, and monotonic versioning (e.g., integer sequence or vector clocks) to detect concurrent edits.

• Concurrency control: choose between pessimistic locking (mutex/RWLock) for simplicity or optimistic concurrency (version compare-and-swap like CAS) for higher throughput and fewer blocking paths.

• Atomic multi-account ops: implement transfers with two strategies — local atomic update using lock ordering to avoid deadlock, or a two-phase commit / compare-and-swap-based retry loop for optimistic multi-row updates.

• Idempotency: require client-supplied idempotency keys for external operations (payments, transfer requests); dedupe by storing recent request keys in a bounded map with result pointers.

• Merge semantics: when merging accounts, apply explicit reconciliation rules (sum balances, preserve transaction history, tombstone source ID) and ensure merges are idempotent and recoverable with a merge log.

• Memory & scale: single-node in-memory systems comfortably handle up to ~1–10M small account objects (~100–1,000 bytes each); beyond that shard by account-id hash or move cold state to Postgres/Redis hybrid.

• Durability tradeoffs: in-memory APIs can persist operation logs or snapshots to Postgres/WAL or append-only files for crash recovery; snapshots every N ops reduce replay time at the cost of extra I/O.

• Failure modes & recovery: design for partial-failure (node crash mid-transfer) using write-ahead logs, committed transaction markers, or idempotent replay that detects completed ops by idempotency key or transaction id.

• Testing & invariants: use property-based tests for conservation of funds, simulated concurrent clients with randomized interleavings, and chaos tests that inject restarts and network partitions.

• APIs & surface: model clear RPCs like CreateAccount, GetBalance(version?), Transfer(from, to, amount, idempotencyKey), MergeAccounts(src, dst, idempotencyKey), with precise success/failure semantics and error codes.

• Metrics & SLOs: track p95/p99 latencies, failed transfers, idempotency cache hit-rate, and reconciliation-backlog size; set alerting thresholds before money is at risk.

• Security & permissions: ensure API enforces authorization at the service layer (who can transfer or merge), and record audit trails for every state-changing operation.

Worked example — Design a Bank Account OOP Simulation with Transfers, Payments, and Merges

First 30 seconds: clarify constraints — single-node or distributed, required durability (in-memory only vs persisted), concurrency expected, and whether negative balances allowed. Declare assumptions: single-process server, strong consistency, and idempotency keys provided by clients.

Skeleton answer pillars: (1) data model — Account {id, balance, version, tombstoned} and Txn {id, from, to, amount, status}; (2) API surface — Create, Deposit, Withdraw, Transfer, Merge with idempotency; (3) concurrency — optimistic version checks with CAS or acquire locks in canonical order (lower-id first) for transfers; (4) durability & recovery — write-ahead log for operations and periodic snapshots; (5) testing & rollout — unit, concurrency, and chaos tests, then canary.

Flaged tradeoff: using coarse-grained locking is simplest and safe but limits throughput; optimistic retries increase throughput but require careful retry/backoff and conflict metrics. Close: if more time, implement pseudo-code for retry loop, show WAL format, and describe sharding strategy and metrics dashboards to detect hot accounts.

A second angle — Design an in-memory cloud storage system

The same responsibilities apply but framed for objects and TTLs instead of money. You'd design an object index, eviction policies, and multi-level storage with LRU + prioritized eviction. Concurrency decisions mirror transfers: multi-object operations (rename/compose) need atomicity. For durability choose snapshot + append-only log or background replication to persistent store. Idempotency and tombstones are critical for object deletes and merges (compose). Performance tradeoffs differ: larger object blobs favor streaming I/O boundaries and background persistence; small metadata updates favor CAS and optimistic concurrency.

Common pitfalls

Pitfall: Forgetting idempotency — naive replays or client retries will double-apply transfers; always require idempotency keys and persist outcomes for a TTL.

Pitfall: Ignoring deadlocks — when locking two accounts, failing to acquire locks in a deterministic global order (e.g., by accountId) leads to deadlock under concurrency; enforce lock ordering or use try-lock with backoff.

Pitfall: Weak failure specs — saying "we'll retry until success" without bounding retries or handling partial commits leads to resource leaks; specify retry limits, backoff, and human-recoverable reconciliation paths.

Connections

Interviewers may pivot to distributed transactions/event sourcing (for multi-node durability), CRDTs (if eventual consistency is acceptable), or cache-invalidation and sharding strategies that scale in-memory services horizontally.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann — deep treatment of consistency models, durability, and replication strategies applicable to stateful in-memory services.

• Redis Transactions and Lua Scripting — pragmatic techniques for atomic operations in an in-memory store context.