Confluent PM Technical Design: Kafka-Style Partitioned, Replicated Data Store

You are interviewing for a product manager role at Confluent. Design a distributed database system, with Kafka as the example to reason from, and explain the key architecture choices and their trade-offs. ```hint Guarantees before boxes Settle what the system promises its users (ordering, durability, throughput, how long data is kept) before deciding how to split and copy data across machines. ``` ### Clarifying Questions - Do you mean a Kafka-style system, an append-only log that many producers write and many consumers read, or a general-purpose database with queries and updates, with Kafka only as a loose example? - Which workload should I design for: high-volume event ingestion, many independent readers, long-term retention, or all three? - Which guarantee matters most to the target customers: strict ordering, no data loss, low latency or low storage cost? - Should I cover how customers run it, self-managed or as a managed cloud service, or only the architecture? ### What a Strong Answer Covers - Requirements and guarantees derived from customer use cases before the architecture - Partitioning and replication, and how they affect ordering, throughput and fault tolerance - The write and read paths, including how readers track their position - Storage architecture, such as shared-nothing local disks versus shared or tiered object storage, judged on cost, latency and elasticity - Failure handling, scaling, and the operational experience and metrics customers see ### Follow-up Questions - A customer needs exactly-once processing from source to sink. What must the system provide, and what does it cost in latency and complexity? - How would moving older data to object storage change cost, latency and the time it takes to scale a cluster? - As the PM for a managed version, which metrics would tell you it is serving customers well?