Kafka partitions, replication, consumer groups, and delivery guarantees

Context

You are designing and evaluating a Kafka-based data pipeline and need to reason about ordering, scalability, and fault tolerance, as well as how to achieve strong delivery guarantees.

Tasks

1. Explain Kafka partitions, replication, and consumer groups:
   • What each is, how it works, and how they interact.
2. Describe how these mechanisms affect:
   • Ordering
   • Scalability
   • Fault tolerance
3. Compare delivery guarantees:
   • At-most-once
   • At-least-once
   • Exactly-once
4. Describe how to implement idempotent, exactly-once processing end-to-end:
   • Within Kafka (topic to topic)
   • When writing to external stores (e.g., databases)
   • Include the key configs, algorithms, and failure handling