Design a payment system with real-time holds, batch settlement and safe retries

Design a payment system. The interviewer started with two money movements: placing a hold on the payer's funds in real time when a payment is made, and settling held payments later in batches. The discussion then turned to how retries are handled without charging anyone twice, and from there to consistency and to what happens in specific failure cases. Expect detailed probing in each area. ### Constraints and Clarifications - Who holds the funds was not specified. If the interviewer leaves it open, assume the system authorizes card or bank payments through an external payment processor (the hold) and captures them later in batches (the settlement), and explain how the design changes if the system instead holds users' balances in its own ledger. - Volumes, settlement frequency and hold expiry rules were not given. Ask, or state your assumptions. ### Clarifying Questions - Is the hold placed through an external processor on a card or bank account, or on a balance the system itself keeps in a wallet ledger? - What is the peak payment rate, and how many payments go into one settlement batch? - When does settlement happen: on a fixed schedule such as daily, or when each order is fulfilled? Can a hold be partially captured or released early? - How long does a hold stay valid before it expires, and what should happen to holds that are never settled? - Are refunds, chargebacks and multiple currencies in scope? ### Part 1 — Real-time hold and batch settlement Design the flow from a payment request to a hold, and from a hold to its settlement in a batch. Include the components, the payment's lifecycle and the data you store. ```hint Two different money movements Treat the hold and the settlement as separate steps with their own outcomes, and list every state a payment can be in between them, including the ones where it never reaches settlement. ``` #### What This Part Should Cover - Components and the synchronous hold path, with its latency target - A payment state machine covering success, decline, release and expiry - The data model, including an auditable record of money movements - How a settlement batch is formed, submitted and confirmed, and how a payment is kept out of two batches ### Part 2 — Retries without double charging Requests are retried by clients, by your own services and by jobs that restart. Explain how the system guarantees that a retried request never holds or settles the same money twice. ```hint What makes two requests the same payment? Decide which identifier marks a request as a retry, where it is stored, and what each hop, including the call to the external processor, does when it sees that identifier a second time. ``` ```hint A timeout is not a failure When a call to the processor times out, you do not know whether it succeeded. Think about what to do next without risking a second charge. ``` #### What This Part Should Cover - Idempotency at the API: what is stored, and how concurrent duplicates and changed payloads are handled - Idempotency toward the processor and in asynchronous workers - Handling outcomes that are unknown after a timeout - Which errors are retried, and how ### Part 3 — Consistency and failure cases Explain where the system needs strong consistency and where eventual consistency is acceptable. Then walk through failure cases, such as a crash at the worst possible moment, and how the system recovers from each. ```hint The external call is outside your transaction Your database and the processor cannot commit together. Consider what you record before and after the external call so that a crash in between can be recovered. ``` #### What This Part Should Cover - What is updated atomically, and how state changes and outgoing events stay in step - Recovering payments stuck between states - Reconciliation against the processor's records - Specific scenarios: a crash after the processor accepted a request, a processor outage, a partially failed batch, an expired hold ### What a Strong Answer Covers - Money semantics (hold versus settlement) clarified before components are drawn - Idempotency enforced by database constraints at every hop, not by read-then-write checks - A clear consistency boundary, with the external processor treated as a separate system that must be reconciled - Concrete failure scenarios with recovery paths, not just "retry on failure" - Auditability: an append-only record of money movements, plus monitoring of stuck states and mismatches ### Follow-up Questions - A customer is refunded after their payment was settled. How does that fit into your state machine and records? - Daily settlement grows to millions of payments and no longer finishes before the next cutoff. What do you change? - Reconciliation finds a payment that the processor captured but your system shows as failed. What happens next, and who is told? - How would you add a second processor for failover without creating duplicate charges?