One Order, Two Charges: Idempotency in SQS and Kafka

A slow email provider, a thirty-second visibility timeout, and a customer billed twice. What at-least-once delivery actually promises and why idempotency is your job, not the broker's.

[One press of the button, two cans in the tray. The everyday face of at-least-once delivery, where a single event fires its side effect more than once.]

---

The Retry That Charged Him Twice

A customer paid once and got billed twice.

I still remember the Slack message — "why are there two charges on this order?" — and the drop in my stomach, because I already suspected it was my fault.

The root cause was mundane, which didn't make it any easier to explain. The handler for a placed order captured the payment, then sent the receipt. The email provider was slow that afternoon. Slow enough that the send dragged past the queue's thirty-second visibility timeout, which is the SQS default and which nobody on the team had ever thought about. The queue assumed the worker had died, and handed the message to a second one. That worker re-ran everything, capture included. Two workers, one order, a card charged twice.

A queue moves the problem downstream; it doesn't solve it. I knew that in the way you know things you've read. I just hadn't believed it hard enough to build for it.

A queue is a promise between two services. I'd never actually read the promise.

I Never Read the Fine Print

The promise is narrower than the word "queue" suggests. The queue holds your message and keeps handing it to a worker until someone acknowledges it — at least once, until it ages out of the retention window. That is the entirety of what you are being offered.

The word once is nowhere in that promise. The queue will deliver the same message five times if that's what it takes; losing it is the one thing it's genuinely afraid of. Delivering it exactly once was never the broker's job — it was mine, down in the consumer, and I'd quietly assumed somebody upstream had it covered.

Here is the sentence I should have written somewhere a reviewer would see it:

Key on order_id. Expect each message at least once. Make the capture idempotent. Send anything failing five times to a dead-letter queue.

Four clauses, absent from the design doc, that would have killed the incident before it started.

The Default That Redelivered My Charge

The system had run fine for months, and that was the dangerous part. Nobody had forced a redelivery, or a slow timeout, or a message the handler couldn't stomach. Months of green dashboards are not evidence of correctness; they're evidence that you haven't met the failure yet.

Our unexamined assumption was that each message arrives once. On SQS Standard it doesn't. If a consumer can run twice, assume it will — the queue's job is to not lose your message, not to count how often it hands it out. Kafka behaves the same way in normal use: a consumer that does the work and dies before committing its offset simply redoes it on restart. Same shape, different logo.

At-least-once wasn't a setting we turned on. It was the contract we'd been building against the whole time, without admitting it.

The word for surviving that contract is idempotency, and it is as unglamorous as it sounds: a handler you can run twice and get the same result as running it once. Without it, at-least-once quietly becomes maybe-twice. And maybe-twice on a payment capture is a refund, a processor fee, and a support ticket you opened for yourself.

The Handler That Charged Twice

Here's the handler that did it, with the bug in plain sight:

def handle_order_placed(msg):

re-runs on redelivery, so it charges again

capture_payment(msg["order_id"], msg["amount"])

also re-runs: phantom stock loss

decrement_inventory(msg["sku"])

slow provider drags past the 30s visibility timeout

send_receipt(msg["email"])

Every line re-runs on redelivery. The capture doesn't know it's the second time. Neither does the decrement — which I missed at first, and which is the same bug wearing a different coat: a double decrement is phantom stock loss, inventory you think you sold and didn't.

The fix everyone reaches for first is a single guard at the top of the function. Seen this message before? Yes? Stop. Don't do that. Mark it seen, crash before the capture runs, and every subsequent redelivery bails on the flag — turning a double charge into no charge at all. You've traded a visible bug for an invisible one.

What holds up is less clever. Make each step idempotent on its own, keyed to the order:

def handle_order_placed(msg):

order_id = msg["order_id"]

Reserve stock before charging, and key every effect on order_id (not message_id),

so a redelivery — or a producer retry carrying a fresh message_id — is safe to replay.

idempotent hold, not a raw decrement

reserve_inventory(order_id, msg["sku"], reservation_id=order_id)

provider charges once, even if called twice

capture_payment(order_id, msg["amount"], idempotency_key=order_id)

left at-least-once on purpose — see below

send_receipt(order_id, msg["email"])

We got to each of those three lines separately, and none of them by accident.

The order flipped. Reserve stock first, then charge, so you never take money for something you can't ship. Some partial failures can't be finished by retrying and have to be undone instead — reversing the charge, releasing the hold. That's a saga — a chain of compensating steps that walk back what already happened — and it's another post. For now, just leave the hatch.

The payment key is the order id, not the message id. This caught a colleague months later. A producer that retries republishes the same order with a fresh message id, and a message-id guard waves it straight through. Key on something the business owns, not something the transport invented.

The receipt is bare on purpose. A redelivery might send it twice, and we decided we could live with that. Match the guard to what the failure costs: a duplicate receipt is a mildly annoyed customer; a lost one prompts an email and you resend. Neither is money. So the capture gets the paranoid treatment and the receipt doesn't.

The deeper fix I put off for months was splitting the three stages apart, so that a slow receipt can't redeliver the payment work at all. We also raised the visibility timeout above our worst-case processing time, which took an afternoon and should have taken an hour on day one.

And then there's the message that never works — failing, retrying, failing, jamming everything behind it. Cap the receive count and route it to a dead-letter queue (DLQ). But a DLQ nobody watches is just a slower graveyard, so give it an owner and a name on the pager.

The duplicate-delivery failure and its fix in one frame. A late acknowledgement triggers a redelivery, and effects keyed on the order id soak up the repeat instead of billing twice.

Once a message fails a set number of times, route it aside rather than letting it block the line forever.

Pick the Shape Before You Pick the Broker

Everyone argues Kafka versus SQS on day one. It should be close to the last thing you decide.

Before you ask why we didn't just switch on SQS FIFO or Kafka's exactly-once semantics: both narrow the window a duplicate can slip through, and neither one makes the charge at your payment provider idempotent. You end up building the same key anyway. The broker narrows the gap; your code closes it.

Underneath the brand names there are three shapes to pick from, and the broker follows from the shape.

A plain queue is work you do once and forget about — a job lands, one worker grabs it, it's gone. That's SQS. Pub/sub is one event that several systems each need their own copy of: billing and the warehouse and analytics all hearing about a placed order independently. That's SNS. A log is durable, ordered and replayable, with each consumer reading at its own pace and events that don't vanish once read. That's Kafka, and it earns its operational cost on the day you replay last month's events into a service that didn't exist last month.

The boundaries are cleaner in a blog post than in a codebase. Teams run Kafka as a plain work queue and regret it slowly. You can bolt fan-out onto SQS by putting SNS in front of it. RabbitMQ with a stream plugin, or Redis Streams, will happily impersonate any of the three depending on how you squint.

So the shapes aren't product categories so much as questions, and the question that does the work is this one: will anyone ever need to replay events that already happened? Yes points at a log. No, and you may have talked yourself into a Kafka cluster you didn't need.

The rest of it — the delivery promise, the dedup key, whether order matters, where dead messages end up — was settled in that one-sentence contract.

The third of those — ordering — is the one that ruins afternoons. Kafka only guarantees order inside a partition. If paid and cancelled for the same order land on different partitions, nothing stops cancelled arriving first, and you spend a morning chasing a cancellation for a payment that can't exist yet. Partition by the thing that needs order, which is the order id, and let the rest scatter.

SQS FIFO holds order within a message group, at a cost you meet under load. Without high-throughput mode, a FIFO queue tops out around three hundred operations a second — a few thousand if you batch. Turn it on and that ceiling applies per message group rather than per queue, so you scale by adding groups. Comfortable for a payment pipeline. Tight for a firehose of analytics events.

What It Actually Cost Us

The direct cost was insultingly small. We reversed the duplicate charge, ate the processor fee twice — once for the capture, once for the reversal — and wrote an apology that took longer than the fix.

The real cost was the retrofit. Idempotency isn't something you bolt onto one handler and call done. The moment it burns you, every other consumer with a side effect becomes the next incident on a timer, so you go and guard all of them. That's a sprint, plus the dedup store and the dead-letter plumbing we should have had on day one.

And then there's a cost that never shows up on an invoice. Once async correctness has burned you, some corner of your brain never switches off again. Every "we'll just fire the email here" now arrives with a small voice asking whether we already did this, and how we would know if we had. You pay that in attention, every single time. On balance I don't think it's a bad trade. It just never appeared in anybody's estimate.

Where This Still Bites, and When to Stay Synchronous

None of this is a law. It's what one double charge taught one team, and your situation will bend it in ways I can't see from here.

But the spine of it holds: don't put an async edge in front of a synchronous job. If B must not happen unless A is durably true — if you cannot ship the goods until the money has cleared — that isn't a message you fire and hope about. Put it in a transaction, or make a direct call and wait. Async is for work that can lag and repeat safely. Any step where "maybe it happened" corrupts your data stays synchronous.

So the rule we ended up with is a dull one, and it has stuck, which is about the highest praise I have for a rule. A new consumer doesn't clear review without a dedup key, a retry limit, a dead-letter queue, and a real name on the pager for when that queue fills. Payments carry a provider idempotency key. Inventory reserves rather than decrements. Email either gets an outbox or tolerates the occasional duplicate. Money gets the paranoid path; a page-view counter can afford to be sloppy.

Here's the part I've made peace with. The double charge isn't gone. It's one careless handler away, permanently. Keys expire, caches get evicted, and six months from now somebody will wire up a consumer having never heard about the contract nobody wrote down. The machine will drop a second can eventually.

The only real job is making sure that when it does, it shows up as a logged line someone catches in a minute — and not as a stranger's bank statement, and a refund three days too late.

---

Postscript, for anyone about to upgrade: Kafka 4.0 dropped ZooKeeper entirely in March 2025, so it's KRaft-only now, and moving an older cluster across still routes through the 3.9 bridge release first.

---

Thank you for reading. If you found this useful, follow PracHub for more practical deep dives on engineering, system design, and interview preparation written to help you build skills that last beyond a single article.