Implement a Kafka-Style Message Partitioner That Supports Adding Partitions

Implement, in an object-oriented style, the part of a Kafka-like message log that places messages into partitions. A topic has a number of partitions, stored as a field of the class. Each new message must be assigned to one of the partitions and stored there. The class must also support adding partitions after messages have already been written. The original prompt was long and only this core was reported, so settle the details below before coding. ### Clarifying Questions - Do messages carry a key? If so, must all messages with the same key land in the same partition? - How should messages without a key be spread across partitions: round-robin, at random, or another way? - Does each partition store its messages with offsets that can be read back, or does the class only decide the partition? - Is the class called from several threads at once? ### Part 1 — Assign new messages to partitions Design the class and implement the method that accepts a new message, chooses its partition and stores it there. The caller should learn where the message was stored. ```hint Same key, same place If keyed messages must stay together, the partition has to be a pure function of the key and the current partition count. Check that the function you pick gives the same answer after a process restart and on another machine. ``` #### What This Part Should Cover - The class's fields and methods, with the partition count as a field - A deterministic rule for keyed messages and a balanced rule for messages without a key - Per-partition storage that preserves arrival order, and validation of inputs ### Part 2 — Add partitions Add a method that increases the number of partitions. Explain what happens to messages already stored, and to the partition chosen for a key that was in use before the change. ```hint What moves Take a key that was written before the change and work out which partition its next message goes to after the count grows. Then ask what a reader that relies on per-key order sees. ``` #### Clarifying Questions for this Part - Is it acceptable for a key's new messages to go to a different partition after partitions are added, or must every key keep its partition? - Can the number of partitions also decrease? #### What This Part Should Cover - What changes for existing keys under the Part 1 rule, and the consequence for per-key ordering - At least one alternative that moves fewer keys, with its cost - Keeping the change safe: no stored message is lost or renumbered, and concurrent sends see a consistent count ### What a Strong Answer Covers - Settles keyed versus keyless messages, the ordering guarantee and the meaning of adding a partition before coding - A small, readable class design with a pluggable partitioning strategy - A stable hash and correct modular arithmetic - An accurate account of which keys move when partitions are added, and a reasoned choice among the options - Thread safety and tests ### Follow-up Questions - Partitions must also be removable. What happens to their messages and to the keys that mapped to them? - One key produces most of the traffic and overloads its partition. What can you do without breaking that key's ordering? - Several producers send concurrently. What do you lock, and which ordering guarantee remains? - How would you add consumers that read each partition from a saved offset?