Interview concept

Financial API Idempotency And Replay Safety

Asked of: Software Engineer

Last updated

What's being tested

This topic tests whether you can design replay-safe APIs in an unreliable distributed environment where clients retry after timeouts, network connections break, and requests may execute more than once. The interviewer is probing for correct use of idempotency keys, durable state, uniqueness constraints, and transactional boundaries—not merely checking whether you can cache a response. Ramp handles financial mutations, so duplicate charges, reimbursements, card authorizations, or transfers are correctness and trust failures, not ordinary availability bugs.

Core knowledge

  • Idempotency means repeating the same logically identified operation has the same externally visible effect as executing it once. GET, PUT, and DELETE are conventionally idempotent; POST usually requires an explicit idempotency-key protocol.

  • An idempotency key should identify one client-intended operation, not merely one HTTP request. Store (tenant_id, endpoint, idempotency_key) under a unique constraint; scoping only by key risks collisions across customers or API resources.

  • Persist a request fingerprint such as SHA-256(canonical_method, path, authenticated_actor, normalized_body). If a key is reused with a different payload, return 409 Conflict or 422 Unprocessable Entity; silently applying the original operation is dangerous.

  • The idempotency record needs durable states such as IN_PROGRESS, SUCCEEDED, and FAILED_RETRYABLE, plus the resource identifier and serialized response. A duplicate arriving during IN_PROGRESS should wait briefly, return a conflict, or poll—not start a second mutation.

  • Enforce uniqueness in the database, not with an application-level “check then insert.” Concurrent requests can both pass the check; Postgres UNIQUE constraints or conditional writes provide the authoritative race-safe boundary.

  • Make the business mutation and idempotency-record transition atomic where possible: insert the key, create the payment, and record the resulting resource in one database transaction. If the external processor is involved, local atomicity alone cannot cover the remote side effect.

  • External calls create an ambiguous outcome: a timeout may mean the processor rejected, accepted, or is still processing the request. Use the processor’s own idempotency key, query-by-client-reference, or reconciliation rather than blindly retrying a charge.

  • Separate exactly-once effects from exactly-once message delivery. Distributed systems generally provide at-least-once delivery; achieve effectively-once business behavior through durable deduplication, deterministic operation identifiers, and idempotent consumers.

  • For asynchronous workflows, persist an outbox event in the same transaction as the state change, then publish it asynchronously. Consumers should deduplicate by event or operation ID and advance a state machine monotonically, tolerating duplicate and reordered messages.

  • Define retention deliberately. A seven-day key lifetime may protect against normal client retries, but a delayed replay after expiration can still duplicate a financial action; retain keys according to settlement, dispute, and compliance windows or reject stale operations using resource state.

  • Response replay must preserve status, headers, and body where practical. Never replay sensitive credentials or secrets; encrypt or redact stored responses, restrict access by tenant, and authenticate every retry before looking up the idempotency record.

  • Idempotency does not mean every retry is safe. A request that failed validation can usually be retried with a new key after correction, while a committed payment should return the original result; document which errors are cached versus retryable.

Worked example

Consider a representative Create Payment API with an Idempotency-Key header, where no specific interview prompt is supplied. In the first 30 seconds, clarify whether “payment created” means a local intent or money movement, what retry window is required, whether clients can submit the same key concurrently, and whether a processor supports idempotency. Organize the answer around request validation and canonical fingerprinting, a durable idempotency state machine, the transaction and uniqueness constraint, and safe integration with the external processor. Store the key scoped to the tenant and operation, atomically create a local payment intent, and return the stored result for later identical retries. Explicitly flag the timeout tradeoff: waiting for an external processor inside a database transaction improves simple reasoning but harms connection utilization, so an asynchronous state machine plus reconciliation may be safer. For a concurrent duplicate, let one request own IN_PROGRESS and make the other poll or receive a retryable response rather than invoke the processor twice. If I had more time, I would define retention, observability, regional failover behavior, authorization checks, and recovery for records stuck in IN_PROGRESS.

A second angle

The same concept applies to an asynchronous reimbursement worker, but the key boundary is message consumption rather than HTTP request handling. Use a stable reimbursement ID as the operation identity, record a completed-effect marker, and make retries consult the marker before issuing another processor call. A crashed worker can leave the local status uncertain, so reconciliation must query the processor or use its idempotency facility. The design must also handle duplicate, delayed, and reordered events without moving a reimbursement backward from COMPLETED to PENDING.

Common pitfalls

Pitfall: “Put the idempotency key in a cache for 24 hours.” A cache can evict entries, lose data during failure, or race under concurrent requests; durable storage and a database uniqueness constraint must protect the financial effect.

Pitfall: Treating every repeated key as success. A key reused with a different body is a client bug or possible attack; compare a canonical request fingerprint and reject the mismatch explicitly.

Pitfall: Claiming that a database transaction makes an external charge exactly once. The processor call can succeed while the local transaction fails, so explain processor-side idempotency, query-based reconciliation, and an explicit UNKNOWN or PENDING state.

Connections

An interviewer may pivot to distributed transactions, the transactional outbox pattern, saga orchestration, or at-least-once message processing. Be ready to discuss Postgres locking and isolation, multi-region consistency, dead-letter recovery, and observability metrics such as duplicate rate, stuck-operation age, and reconciliation lag.

Further reading

Related concepts