Design a Reliable Notification System

Design a notification system. Explain the storage choices, recipient fanout strategy, behavior under high traffic, and reliability model. Include the operational trade-offs and maintenance concerns that matter after the initial design is deployed. ### Constraints and Clarifying Questions - Clarify notification sources, channels, recipient counts, arrival patterns, latency goals, and retention needs. No specific channels or numerical targets are supplied. - Establish whether recipients are fixed when an event is accepted or evaluated later. - Define what accepted, attempted, delivered, and failed mean for each chosen channel. - Ask about preferences, ordering, duplicate tolerance, and whether old notifications can become irrelevant. ### Part 1 — Accept and Store Notification Work Describe the path from a producer's request to durable notification work. Choose storage for event data, recipient or subscription information, and delivery progress. #### What This Part Should Cover - Stable identities, validation, durable acceptance, and retry handling. - Storage choices connected to access patterns and consistency needs. - A recoverable link between accepted events and work that must be dispatched. ### Part 2 — Fan Out and Handle High Traffic Explain how one event reaches its recipients and how the design behaves under large audiences, hot partitions, bursts, and a constrained downstream channel. #### What This Part Should Cover - An explicit fanout strategy and the point at which recipient membership is determined. - Bounded batching, partitioning, and backpressure. - Scheduling or isolation choices that prevent one large workload from blocking unrelated notifications. ### Part 3 — Deliver Reliably and Maintain the System Explain failure recovery, delivery-state updates, and how operators identify and repair incomplete work. Describe how the design can evolve without breaking existing producers or losing accepted notifications. #### What This Part Should Cover - Retry and duplicate-handling behavior at the external delivery boundary. - Clear state semantics and reconciliation of ambiguous outcomes. - Useful monitoring, retention, replay, and schema-evolution policies. ```hint Trace one accepted notification Follow one logical notification across a process crash, a large fanout, and an uncertain channel response. Identify the durable identity and evidence used at each boundary. ``` ### What a Strong Answer Covers - A coherent data and processing model tied to the stated workload. - Scalable fanout that respects channel limits and preserves recoverability. - Operational reasoning about failures, duplicate effects, and long-term changes. ### Follow-up Questions - What changes when an event targets a very large audience rather than a single recipient? - How would you replay a failed batch without re-notifying recipients whose delivery already completed? - Which records must survive long enough to support the chosen retry and reconciliation guarantees?