Explain how Kafka works

Prompt

Explain how Apache Kafka works at a high level and then in more detail.

Cover at least:

• Core entities: broker , topic , partition , producer , consumer , consumer group
• How messages are stored (append-only log) and how offsets work
• How Kafka achieves scalability (partitioning) and fault tolerance (replication, leader/follower)
• Delivery semantics: at-most-once , at-least-once , and what is required for effectively/exactly-once processing
• Ordering guarantees (what is ordered and what is not)
• What happens during failures (broker failure, consumer failure, rebalance)

You may assume a typical multi-broker Kafka cluster.