Allocating Partitions and Consumers for a Kafka-Style Event Topic

In a hiring-manager round, a discussion of the team's data systems turns to how partitions and consumers should be allocated in a partitioned message log. The report names only this topic, so this version assumes a Kafka-style log: producers write keyed events to a topic that is split into partitions, and a group of consumer instances processes them. Explain how you would allocate partitions and consumers for such a topic. ### Clarifying Questions - Must events with the same key, for example the same order or customer, be processed in order? - What peak event rate and event size must the topic handle, and how far behind may consumers fall? - Do other consumer groups read the same topic for different purposes? - Is processing idempotent, or must each event take effect exactly once? ### Part 1 — How many partitions and how many consumers How do you choose the number of partitions for the topic and the number of consumer instances that process it? What happens when the two numbers differ? ```hint What one partition allows Think about what a single partition guarantees, and what that implies for how many consumers in one group can work on it at the same time. ``` #### What This Part Should Cover - Partitions as the unit of both ordering and parallelism, and how keys map to partitions - Sizing the partition count from throughput, with headroom for growth - The costs of too many partitions and of adding partitions later - What happens with fewer, equal or more consumers than partitions ### Part 2 — Assigning partitions and rebalancing How are partitions assigned to the consumers of a group, and what happens when a consumer is added, removed, restarted or becomes slow? ```hint Follow one crash Walk through a single consumer crashing: who notices, which partitions move, and which events might be processed twice. ``` #### What This Part Should Cover - Assignment strategies and their trade-offs - What triggers a rebalance, and how processing is affected while it runs - Offset commits and duplicate processing when partitions change owner - Ways to avoid needless rebalances during deployments ### What a Strong Answer Covers - Correct mechanics of partitions, keys and consumer groups - Sizing arithmetic with explicit headroom and growth - Behavior during failures and deployments, including duplicate processing - Handling skew from hot keys and uneven partitions - Monitoring consumer lag per partition and acting on it ### Follow-up Questions - One customer produces a large share of all events, so its partition lags while the others are idle. What do you do, and what do you give up? - You need to double throughput, but the group already has as many consumers as the topic has partitions. What are your options? - Consumers restart during every deployment and each restart pauses processing. How do you reduce the impact? - A second team wants to read the same events at a very different processing speed. How do you set that up?