Idempotent Kafka Consumer: Duplicate Writes, Offset Commits and a Second Database

A backend service consumes messages from a Kafka topic. Processing a message writes a record to a database, and downstream business logic continues from that record; nothing is returned to an external client. The same message can be processed more than once, and in the interviewer's scenario the same message is processed by several threads at the same time. How do you guarantee idempotent processing under this concurrency? The interviewer then pushed on what happens when the write finds the record already present, and on keeping a second database consistent. ### Clarifying Questions - Does every message carry a stable unique identifier, or must one be derived from its business content? - How does one message reach several threads: a worker pool inside one consumer, redelivery after a crash or rebalance, or duplicate sends by the producer? - Does anything wait for the result of processing, or is the result consumed only by downstream logic? - Must the second database agree with the first at every instant, or is a short delay acceptable? ### Part 1 — Idempotency under concurrent processing Explain where duplicate processing comes from in a Kafka consumer, and design the processing so that each message's effect is applied exactly once, even when two threads handle the same message at the same moment. ```hint Where the decision can be atomic Two threads that each check "already processed?" and then write can both see "no". Find a place where the check and the write become a single atomic step. ``` #### What This Part Should Cover - The sources of duplicate delivery: at-least-once consumption, crashes before the offset commit, rebalances, producer retries - A stable idempotency key and an atomic insert or claim keyed on it - Why a check-then-write in application code fails under concurrency, and what partitioning by key does and does not solve ### Part 2 — The write finds the record already there A second thread processing the same message reaches the database write and finds that the record already exists. You could throw a duplicate exception; the interviewer suggested returning the existing data instead. Which do you choose, and why? If you throw, does the consumer's offset stop moving? ```hint A duplicate is not a failure Decide whether finding the record means "this message failed" or "this message's work is already done", and let that decide what happens to the offset. ``` ```hint The first attempt may have stopped halfway Consider a thread that wrote the record and then crashed before it handed the work downstream. What does the redelivered message find? ``` #### What This Part Should Cover - Treating the duplicate as a successful outcome, and when throwing and returning the existing data are equivalent - How offsets advance when the duplicate is caught, including commits when several threads finish out of order - Recovery when the first attempt stopped between the database write and the downstream step ### Part 3 — Keeping a second database in sync Processing must also update a second database asynchronously. How do you keep the two databases consistent? If you publish a message to trigger the second write and publishing fails, what happens? ```hint One commit for two intentions Ask what single commit could record both "the first database changed" and "the second database still needs to change". ``` #### What This Part Should Cover - Why writing to two systems one after the other leaves gaps on partial failure - A way to tie the follow-up message to the first database's commit, and its delivery guarantee - Idempotency and ordering on the second database's side, and when a distributed transaction is worth its cost ### What a Strong Answer Covers - A precise statement of the delivery guarantee and of where duplicates arise - One idempotency key used consistently from consumption through every write - Offset handling that neither loses messages nor stalls a partition behind a duplicate - A consistency approach for the second database that survives a crash at every step - Monitoring and reconciliation to detect drift, and trade-offs between simplicity, latency and guarantee strength ### Follow-up Questions - The consumer commits offsets automatically on a timer. Which failures now lose messages, and which duplicate them? - How long must idempotency keys be kept, and what happens if a duplicate arrives after its key is purged? - Would Kafka's idempotent producer and transactions remove the need for database-side deduplication here? Why or why not? - The second database starts lagging the first by an hour. What does the business see, and how do you detect and recover?