"Just Make It Idempotent" Is Half an Answer in a System Design Interview

Most candidates stop at the key. The interviewer is waiting for the transaction, the in-flight reservation, and the outbox.

A key that opens nothing on its own. The vault is the transaction, which is the whole point of the reframe.

The payment retries. Now what?

The charge request times out with no 200, so the client retries, exactly as it's supposed to.

Somewhere a senior engineer says "just make it idempotent" and everyone nods like that settled it. It didn't. The phrase names a goal and stops there. It doesn't tell you where the key lives, or what happens when two retries hit at the same instant. The real question is what happens to the second apply, and most services get it wrong in a way that surfaces as a double charge three weeks later.

I've shipped the wrong version of this myself. Let's kill the myth first, then build the thing that holds.

Exactly-once delivery does not exist

Most architecture diagrams hide this, but the truth is simpler, and harder to accept: you cannot deliver a message exactly once across a network that can drop packets.

Picture the sender waiting for an ack that never comes back. It can't tell two cases apart: the message never arrived, or it arrived and the ack got lost coming home. From the sender's side the two look identical, so it has to guess.

So it picks one of two behaviours. Retry and risk a duplicate, which is at-least-once. Or stay quiet and risk losing the message, which is at-most-once. That's the choice. There is no third door. Exactly-once delivery isn't possible on an unreliable network.

You don't get exactly-once delivery. You get at-least-once delivery plus idempotent processing, and to anyone reading the data, the two are indistinguishable.

Accept duplicates at the wire, and make the processing absorb them so a duplicate has no extra effect. The visible state matches a world where the message arrived once.

PUT is safe. POST /orders ten times is ten orders.

Some operations are idempotent for free. PUT /users/123 {balance: 100} sets a value. Run it once or ten times and the balance is 100. Setting an absolute value is naturally safe.

The dangerous ones do rather than set. POST /orders creates a row every call, so ten retries means ten orders and ten emails. UPDATE accounts SET balance = balance - 100 is a delta, so ten applies subtract a thousand. Of course, you can wrap even a delta in the deduplication pattern we're about to build: a unique transaction reference the database rejects on duplicate.

For those, "make it idempotent" isn't a property you have. It's a mechanism you build: a deduplication key with something durable behind it.

The key and the effect commit together, or you have nothing

Here's where teams think they're done and aren't. The client generates a unique key per logical operation, a UUID sent as an Idempotency-Key header, the way Stripe does it. The server has to land that key and the effect it guards in the same transaction, with a database uniqueness constraint as the arbiter.

Miss that and you write the classic check-then-act race:

-- retry #2 arrives while retry #1 is still mid-flight

SELECT id FROM payments WHERE idem_key = $1;   -- returns nothing for BOTH requests

-- both think they're first, both charge the card, both insert

INSERT INTO payments (idem_key, amount) VALUES ($1, 5000);

Two concurrent retries both read "no row," so both do the work. A separate check step is a window, and windows get raced. Stop checking and let the constraint decide:

BEGIN;

-- the effect and the key commit as one unit; UNIQUE(idem_key) is the arbiter

INSERT INTO payments (idem_key, amount) VALUES ($1, 5000)

ON CONFLICT (idem_key) DO NOTHING;   -- the second apply is a no-op, not a second charge

COMMIT;

No read-then-write. The database resolves the collision atomically and one insert wins. On conflict, you read the existing row and return the same response the first call produced. That's why Stripe stores the response body next to the key, not just the key: a retry gets the identical answer, not a fresh run.

If you only remember one thing, let it be this:

If the key isn't written in the same transaction as the effect, you didn't build idempotency. You built a suggestion.

The best-effort cache trap, and the two-phase fix

I shipped this exact version once. Wrote "idempotency: done" on the ticket and closed it. It was a cache wearing a correctness costume.

if redis.get(idem_key):          # Redis down? key expired? -> check passes, double charge

return cached_response

charge_card(amount)              # the effect happens OUTSIDE any shared transaction

redis.setex(idem_key, 3600, response)  # process dies on this line -> the retry re-charges

Every failure mode here is silent. The key lives in a side cache with a TTL, the effect lives in your database, and nothing binds them. Redis blips, or the key expires early, or the process dies between the charge and the setex, and a duplicate walks through. If you're de-duping analytics events, a cache with a TTL is probably fine. For payments, it's a time bomb waiting for the sort of Saturday morning you don't want to be on call.

There's a second problem the cache hides. What if the first request is still running when the retry lands? You need a two-phase reservation. Reserve the key the moment the request arrives, in an in_progress state:

-- phase 1: reserve. if this row already exists, you're a retry, not the first caller.

INSERT INTO idem_keys (key, status) VALUES ($1, 'in_progress');  -- UNIQUE(key)

-- phase 2: do the work, then in ONE txn flip status to 'done' and store the response body

If the reservation conflicts and the row is in_progress, the retry returns 409 or waits rather than starting a second run. If it's done, you return the stored response. Then the decision most tutorials skip: when the operation fails after you reserved the key, what happens to that row? Leave it in_progress and every retry is stuck. Delete the reservation if the client can safely retry fresh. Mark it failed with a reason if you need an audit trail and the client must check why first. Make that call on purpose, and you've turned a code snippet into a design that doesn't leak when the pressure goes up.

The interviewer ladder

Interviewers use this topic as a filter. The opening question checks if you've heard the phrase. The later ones check if you've been paged because you missed the fine print.

It starts easy. "How do you handle a retried payment?" Then it escalates: where's the key stored, what arbitrates a concurrent collision, what happens mid-flight, how do you publish atomically with the write. Each rung drops someone who only memorised the phrase.

Three quick follow-ups come up every time.

Client or server generates the key? The client. It has to survive the client's own retries, so it can't be minted server-side per request.

How long do you keep keys? Long enough to cover the client's full retry horizon, then archive. A 24-hour retry against a purged key re-executes.

Still need consumer idempotency with a FIFO queue? Yes. The dedup window is finite and replays outlive it.

Then the big one: doesn't Kafka or SQS already handle this?

Not on its own. Kafka's idempotent producer is on by default since 3.0, and it stops a producer's own retries from writing the same record twice to a partition. That's the whole promise. The term "exactly-once semantics" sounds bigger than what you actually get. Underneath, it's the same thing you built: at-least-once delivery plus atomic processing, and only inside a Kafka application, not end-to-end. SQS FIFO gives you a deduplication ID with a 5-minute window; a duplicate outside that window is processed as a fresh message. Bounded, not permanent.

And when you need to change your database and publish an event, you hit the dual-write problem: you can't commit to Postgres and Kafka in one transaction. So don't. Write the event to an outbox table in the same transaction as the state change. A separate relay reads new outbox rows and publishes them after the commit, not alongside it, so the event ships only if the state was durably written. Pair that with idempotent consumers and you're back to effectively-once, built from honest at-least-once parts.

What it costs, and when not to bother

The right version isn't free. The two-phase reservation adds a write and a state machine to every guarded operation. The in-progress-versus-failed choice is a human call nobody makes for you. The outbox adds a relay to run, monitor, and get paged on. And the reservation round-trip adds latency to every request, even though the duplicate it guards against might only strike during network trouble or timeouts.

Wave that cost off on payments, though, and the bill isn't hypothetical: a queue of "why was I charged twice" emails, a reconciliation spreadsheet nobody volunteers for, and the trust you lose when a customer sees a double debit. A duplicate that slips through lands on finance's desk first, then bounces straight back into your on-call shift. The cache version I shipped is the exact shape that produces that bill.

So calibrate to blast radius. Running an outbox for a fire-and-forget analytics event is over-engineering; a cache with a TTL is fine there. Do the full build when a duplicate means money moved twice, an email went out twice, or a downstream system can't tell the difference.

There's no tidy ending here. Add a second datastore, or a cache, or another queue, and the thing you thought was airtight starts to leak a little, and someone has to catch it before a customer does. You never really tick "idempotent" and walk away. Something new gets bolted on, and you go back to check the guarantee still holds, because you can't assume it does.

If you want to go deeper, the pieces on at-least-once delivery, the atomicity race, and why exactly-once trades away availability are where the scars show.

If you're preparing for a system design interview, or just trying to survive your next production outage, followPracHubfor more engineering deep dives.