Explain Kafka Reliability and Offset Recovery
Company: Illumio
Role: Software Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Onsite
You are using Kafka to process events in a distributed system.
Explain the following:
1. How do partitions affect ordering, parallelism, and scalability?
2. What does `at-least-once` delivery mean in Kafka-based systems?
3. Suppose processing for offset `2` fails, but a later offset such as `4` is already committed. What went wrong, what is the consequence, and how should the system recover or prevent this situation?
4. How would you tune Kafka and its consumers/producers to achieve both high throughput and high reliability?
Quick Answer: This question evaluates understanding of distributed messaging and event-processing concepts, specifically partitioning effects on ordering, parallelism and scalability, delivery semantics like at-least-once, offset management and recovery, fault tolerance, and operational tuning for throughput and reliability.