Design a Secure, Highly Available Real-Time Event Messaging Platform and Defend It

This is a take-home system design followed by a live review. You receive a written set of requirements, design a real-time event messaging system on your own time, and submit the design. Later you meet engineers from the team for an extended session in which they walk through your submission, ask questions, raise concerns, suggest alternatives, and sometimes change or add requirements you did not plan for. The requirements document from the interview is not reproduced here. For practice, assume a platform that internal services use to publish events, which other services consume in near real time, at a company where security and availability are the two properties the design must protect above everything else. Weigh every trade-off in terms of the people and systems that depend on the platform. ### Constraints and Clarifications - Security and availability are first-class requirements, not a closing section. - The review rewards collaboration: explaining your reasoning, taking suggestions seriously, and changing your mind when a reviewer has a better option, rather than defending every choice. - Expect requirements to move during the review. ### Clarifying Questions - Who publishes and who consumes: internal services only, or also external clients or devices? - What event rates and sizes are expected at peak, and how bursty is the traffic? - What end-to-end latency counts as real time for the consumers? - Which delivery guarantee is required: at-most-once, at-least-once, or effectively exactly-once? - Does ordering matter, and if so, per what key? - How long must events be retained, and do consumers need to replay history? - Is the platform shared by several tenants or teams that must be isolated from one another? ### Part 1 — The design you submit Produce the design you would hand in: requirements and assumptions, capacity estimates, the architecture and data flow from publish to delivery, the event and subscription data model, the APIs, and how the design meets its security and availability goals. Note the alternatives you rejected and why. ```hint Write it for the review The engineers will read your document before they talk to you; decide which decisions are most likely to be challenged and make your reasoning for those explicit on the page. ``` #### What This Part Should Cover - Delivery semantics, ordering and retention chosen deliberately and justified - A durable, replicated, partitioned core with a clear publish path and consume path - Concrete security controls: identity, authorization, encryption, isolation and auditing - Availability through replication, failover and graceful degradation, with recovery targets ### Part 2 — Defending and adapting the design in review Now the reviewers push back. For practice, respond to three challenges as you would in the room: a reviewer argues that a component you designed should be replaced by an existing managed product; a reviewer asks what happens to producers and consumers when an entire region goes down; and the team adds a new requirement that some events must never be processed twice. For each, show how you evaluate the point, whether you concede or hold your position, and how the design changes. ```hint Restate before you respond Before arguing for or against a suggestion, say back what problem the reviewer is worried about; you may agree with the concern even if you do not agree with the proposed fix. ``` #### What This Part Should Cover - Engaging with the concern behind a suggestion before judging the suggestion itself - Conceding gracefully when an alternative is better, and holding firm with evidence when it is not - Updating the design concretely for a changed requirement, including what it costs - Keeping a record of the decisions and open questions from the session ### What a Strong Answer Covers - Trade-offs framed in terms of the platform's users and how much failure each can tolerate - Security designed into every path, not added only at the edge - Availability analysis that names what fails, what users see and how quickly it recovers - Honest acknowledgment of limits and of what would still need validation - Collaborative behavior in the review: listening, conceding and adapting ### Follow-up Questions - One tenant starts publishing ten times its normal volume. How does the platform protect everyone else? - How would you rotate the keys that encrypt stored events without downtime? - A consumer falls hours behind. How do you detect it, and what are its options for catching up? - How would you migrate existing producers onto this platform without losing or duplicating events?