Design Notifications for One Million Internal Producer Services

Design a notification system that can accept notifications from one million internal services. ### Requirements and Constraints The supplied scale is the number of producing services, not a message-per-second rate. Clarify message volume, bursts, destinations, delivery semantics, and latency requirements before sizing the data path. No delivery channel, recipient count, or external provider is mandated. Explain how a large population of services integrates with the system, how notifications are accepted and routed, and how overload or delivery failure is handled. ### Clarifying Questions - Are all one million services active concurrently, and what is the distribution of notification rates across them? - Are notifications delivered to people, other services, or both, and through which channels? - Which notifications may be delayed, duplicated, expired, or dropped? - What isolation, quotas, ordering, and access policies are required between producers? ```hint Separate producer count from traffic volume A million registered services can produce a small average load or an extreme burst. The integration and identity problem is large even before the actual message rate is known. ``` ### What a Strong Answer Covers - Producer registration, identity, schema or API contracts, and manageable client integration at the supplied service count. - Durable acceptance and a decoupled routing/delivery path when reliability requires it. - Capacity planning from actual rates, fanout, payload size, and burst behavior rather than the producer count alone. - Quotas, backpressure, and isolation that prevent a noisy producer from monopolizing delivery. - Retry, deduplication, expiration, and observable failure semantics suited to the chosen channels. - A distinction between metadata/control-plane scale and notification data-plane throughput. ### Follow-up Questions 1. How would you handle a coordinated burst from many services during an infrastructure incident? 2. Why might creating a separate queue or persistent connection for every producer be expensive? 3. What should a producer infer from an accepted response if the final delivery is asynchronous?