Explain Kafka and Message-Queue Delivery Semantics

Explain how a Kafka-style log differs from a traditional work queue. Then design the producer and consumer behavior for an event that must be processed reliably despite retries, consumer crashes, and duplicate delivery. ### Constraints & Assumptions - Assume the broker can durably replicate records but end-to-end side effects occur in another system. - Ordering is needed per entity, not necessarily across all entities. - The answer must distinguish broker acknowledgement from completed business processing. ### Clarifying Questions to Ask - Can consumers process an event more than once safely? - What ordering key maps related events to the same partition? - How much replay history is required? ### What a Strong Answer Covers - Partitions, offsets, consumer groups, retention, and the difference between log replay and queue deletion. - Producer durability and idempotence choices. - Commit timing and the at-most-once versus at-least-once trade-off. - Idempotent side effects or an inbox/outbox pattern for practical end-to-end reliability. - Poison-message handling, backpressure, lag, rebalancing, and observability. ### Follow-up Questions - How would you preserve per-customer ordering while increasing throughput? - What happens if the database commit succeeds but the offset commit fails? - When would a conventional queue be simpler than a retained log?