Financial API Idempotency And Replay Safety
Asked of: Software Engineer
Last updated
What's being tested
This topic tests whether you can design replay-safe APIs in an unreliable distributed environment where clients retry after timeouts, network connections break, and requests may execute more than once. The interviewer is probing for correct use of idempotency keys, durable state, uniqueness constraints, and transactional boundaries—not merely checking whether you can cache a response. Ramp handles financial mutations, so duplicate charges, reimbursements, card authorizations, or transfers are correctness and trust failures, not ordinary availability bugs.
Core knowledge
-
Idempotency means repeating the same logically identified operation has the same externally visible effect as executing it once.
GET,PUT, andDELETEare conventionally idempotent;POSTusually requires an explicit idempotency-key protocol. -
An idempotency key should identify one client-intended operation, not merely one HTTP request. Store
(tenant_id, endpoint, idempotency_key)under a unique constraint; scoping only by key risks collisions across customers or API resources. -
Persist a request fingerprint such as
SHA-256(canonical_method, path, authenticated_actor, normalized_body). If a key is reused with a different payload, return409 Conflictor422 Unprocessable Entity; silently applying the original operation is dangerous. -
The idempotency record needs durable states such as
IN_PROGRESS,SUCCEEDED, andFAILED_RETRYABLE, plus the resource identifier and serialized response. A duplicate arriving duringIN_PROGRESSshould wait briefly, return a conflict, or poll—not start a second mutation. -
Enforce uniqueness in the database, not with an application-level “check then insert.” Concurrent requests can both pass the check;
PostgresUNIQUEconstraints or conditional writes provide the authoritative race-safe boundary. -
Make the business mutation and idempotency-record transition atomic where possible: insert the key, create the payment, and record the resulting resource in one database transaction. If the external processor is involved, local atomicity alone cannot cover the remote side effect.
-
External calls create an ambiguous outcome: a timeout may mean the processor rejected, accepted, or is still processing the request. Use the processor’s own idempotency key, query-by-client-reference, or reconciliation rather than blindly retrying a charge.
-
Separate exactly-once effects from exactly-once message delivery. Distributed systems generally provide at-least-once delivery; achieve effectively-once business behavior through durable deduplication, deterministic operation identifiers, and idempotent consumers.
-
For asynchronous workflows, persist an outbox event in the same transaction as the state change, then publish it asynchronously. Consumers should deduplicate by event or operation ID and advance a state machine monotonically, tolerating duplicate and reordered messages.
-
Define retention deliberately. A seven-day key lifetime may protect against normal client retries, but a delayed replay after expiration can still duplicate a financial action; retain keys according to settlement, dispute, and compliance windows or reject stale operations using resource state.
-
Response replay must preserve status, headers, and body where practical. Never replay sensitive credentials or secrets; encrypt or redact stored responses, restrict access by tenant, and authenticate every retry before looking up the idempotency record.
-
Idempotency does not mean every retry is safe. A request that failed validation can usually be retried with a new key after correction, while a committed payment should return the original result; document which errors are cached versus retryable.
Worked example
Consider a representative Create Payment API with an Idempotency-Key header, where no specific interview prompt is supplied. In the first 30 seconds, clarify whether “payment created” means a local intent or money movement, what retry window is required, whether clients can submit the same key concurrently, and whether a processor supports idempotency. Organize the answer around request validation and canonical fingerprinting, a durable idempotency state machine, the transaction and uniqueness constraint, and safe integration with the external processor. Store the key scoped to the tenant and operation, atomically create a local payment intent, and return the stored result for later identical retries. Explicitly flag the timeout tradeoff: waiting for an external processor inside a database transaction improves simple reasoning but harms connection utilization, so an asynchronous state machine plus reconciliation may be safer. For a concurrent duplicate, let one request own IN_PROGRESS and make the other poll or receive a retryable response rather than invoke the processor twice. If I had more time, I would define retention, observability, regional failover behavior, authorization checks, and recovery for records stuck in IN_PROGRESS.
A second angle
The same concept applies to an asynchronous reimbursement worker, but the key boundary is message consumption rather than HTTP request handling. Use a stable reimbursement ID as the operation identity, record a completed-effect marker, and make retries consult the marker before issuing another processor call. A crashed worker can leave the local status uncertain, so reconciliation must query the processor or use its idempotency facility. The design must also handle duplicate, delayed, and reordered events without moving a reimbursement backward from COMPLETED to PENDING.
Common pitfalls
Pitfall: “Put the idempotency key in a cache for 24 hours.” A cache can evict entries, lose data during failure, or race under concurrent requests; durable storage and a database uniqueness constraint must protect the financial effect.
Pitfall: Treating every repeated key as success. A key reused with a different body is a client bug or possible attack; compare a canonical request fingerprint and reject the mismatch explicitly.
Pitfall: Claiming that a database transaction makes an external charge exactly once. The processor call can succeed while the local transaction fails, so explain processor-side idempotency, query-based reconciliation, and an explicit
UNKNOWNorPENDINGstate.
Connections
An interviewer may pivot to distributed transactions, the transactional outbox pattern, saga orchestration, or at-least-once message processing. Be ready to discuss Postgres locking and isolation, multi-region consistency, dead-letter recovery, and observability metrics such as duplicate rate, stuck-operation age, and reconciliation lag.
Further reading
-
Stripe API: Idempotent Requests — practical treatment of key reuse, replayed responses, and parameter mismatches.
-
Martin Kleppmann, Designing Data-Intensive Applications, Chapters 8–11 — rigorous coverage of retries, distributed transactions, streams, and consistency.
-
Pat Helland, “Life Beyond Distributed Transactions” — useful framing for application-level business operations and recovery.
Related concepts
- Reliable Payment Processing And IdempotencySystem Design
- Fintech Ledger Idempotency and Money Movement
- Idempotent API DesignSystem Design
- API Idempotency And Concurrency ControlSystem Design
- Transactional Consistency, Idempotency, And Exact ContractsSystem Design
- Reliable Distributed Systems And IdempotencySystem Design