System Design: At-Least-Once Notification System

Context

Design a multi-channel notification platform (email, SMS, push, in-app) that guarantees at-least-once delivery. Duplicates are possible, so the system must be idempotent and provide a good user experience without showing duplicates. Assume high throughput (e.g., thousands of notifications per second), multi-tenant services, and retries across transient failures.

Requirements

1. High-level architecture and flow that provides at-least-once delivery.
2. Producer interface (API/SDK) and consumer/worker interface (message schema, ack/nack semantics).
3. Deduplication strategy: keys, storage, TTL, and uniqueness constraints.
4. Retry policy with exponential backoff and jitter; failure handling and dead-letter queues.
5. Out-of-order handling across retries and parallelism; ordering guarantees per key.
6. User-facing idempotency: ensure no duplicate notifications are displayed to users across channels.

Deliverables

• Define producer API (fields, idempotency, responses) and consumer message interface (fields, ack/nack).
• Specify deduplication key design and how/where it is stored.
• Describe retry algorithm (exponential backoff with jitter), limits, and DLQ behavior.
• Describe how to handle out-of-order events (partitioning keys, sequence numbers, buffering windows).
• Describe user-facing idempotency for UI and downstream providers (e.g., collapse/replace semantics).
• Call out assumptions and trade-offs.