## Design an Alert Notification Service for Downstream Consumers Design an alert-notification system whose recipients are downstream services rather than end users. Producers publish an alert with a stable ID, type, payload reference, severity, and creation time. Subscription rules determine which downstream service endpoints should receive it. A delivery can fail temporarily or permanently, and traffic can arrive in bursts. ### Constraints & Assumptions - A producer should not wait for every downstream delivery to finish. - One slow subscriber must not block unrelated subscribers. - Network delivery is at least once; subscribers may receive a duplicate and must have a way to recognize it. - Retry behavior must be bounded and observable. - Payloads may be sensitive, so routing and access checks occur before delivery. ### Part 1 — Define Ingestion, Subscriptions, and Delivery State Specify the producer API or event contract, subscription model, and durable state needed to track one alert across several downstream services. #### What This Part Should Cover - Stable alert and per-destination delivery identifiers. - Validation, authorization, and durable acceptance before acknowledging the producer. - Subscription versioning so a delivery can be explained later. - Independent delivery state for each matched subscriber. ```hint Expand one alert into destination-specific work A single accepted alert can succeed for one subscriber and fail for another, so one global status is not enough. ``` ### Part 2 — Implement Retry and Failure Handling Explain the full lifecycle of a delivery attempt. Distinguish timeout, rate limit, server error, authentication failure, invalid endpoint, and subscriber rejection. Include idempotency, backoff, retry scheduling, and the terminal path. #### What This Part Should Cover - Per-attempt timeouts and error classification. - Exponential backoff with jitter and a retry budget. - A dead-letter or terminal-failure workflow with replay controls. - Safe handling of “request accepted but response lost.” ```hint Classify before retrying The same retry policy should not be applied to a rate limit, a malformed URL, and an authorization failure. ``` ### Part 3 — Scale and Operate the System Show how queues, workers, partitioning, backpressure, and observability keep the system responsive during a burst or a subscriber outage. Address whether ordering is needed and at what key. #### What This Part Should Cover - Queue isolation or fair scheduling that contains a noisy or failing subscriber. - A partition key chosen from the actual ordering requirement. - Backpressure, admission control, and retention limits. - Lag, latency, success, retry, terminal-failure, and duplicate-delivery signals. ```hint Choose the narrowest useful ordering key Global ordering reduces parallelism; ask whether only alerts for one resource and subscriber need to remain ordered. ``` ### What a Strong Answer Covers - Separates durable ingestion from asynchronous fan-out and delivery. - Makes retry mechanics concrete instead of treating a retry queue as a complete design. - Provides idempotency and reconciliation for ambiguous network outcomes. - Scales by destination while preserving only the ordering the product requires. - Gives operators enough state to diagnose and safely replay failures. ### Follow-up Questions 1. How would you stop one unavailable subscriber from consuming every worker slot? 2. When should an alert be considered successfully processed: after ingestion, after any delivery, or after all required deliveries? 3. How would a subscriber prove that two requests are retries of the same alert? 4. What changes if a subscription is edited while old alerts are still queued?