Design a global notification service
System Design: Globally Distributed Notification Service
Context
You are designing a multi-tenant notification platform that delivers real-time and scheduled messages (email, SMS, push) to tens of millions of users worldwide. The system must comply with regional data residency and privacy regulations, support high availability across multiple regions, and provide strong operational controls (idempotency, deduplication, rate limiting, retries, monitoring).
Assume:
-
Users and data are partitioned by region (e.g., US, EU, APAC) with strict residency for PII.
-
The platform offers APIs to trigger individual and bulk notifications and manage templates and user preferences.
-
Peak traffic can spike rapidly (e.g., incident alerts, promotions).
Requirements
Design a system that addresses the following:
-
APIs and Data Models
-
Define REST APIs for:
-
Sending real-time and scheduled notifications (single and bulk)
-
Managing templates and variables
-
Managing user preferences and subscriptions
-
Retrieving message status and delivery receipts
-
Specify core data models (Message, DeliveryAttempt, Template, Campaign/Job, UserPreference, IdempotencyRecord, RateLimitBucket).
-
Deduplication, Idempotency, and Rate Limiting
-
Describe how to prevent duplicate sends across retries and concurrent requests.
-
Provide idempotency strategy at API and worker levels.
-
Define rate limiting scopes (per-user, per-tenant, per-channel, per-provider) and algorithms.
-
Storage and Queueing Layers
-
Choose storage for:
-
Control-plane metadata (tenants, templates, campaigns)
-
Regional data-plane (messages, attempts, user preferences)
-
Caching (idempotency keys, rate limiter state)
-
Object/blob storage (large templates, assets)
-
Choose queueing/streaming for fan-out, ordering, retries, scheduled delivery, and DLQs.
-
Worker Orchestration, Retry/Backoff, Ordering
-
Describe worker topology and autoscaling.
-
Define retry/backoff policies and DLQ handling by failure type.
-
Specify ordering guarantees (e.g., per-user per-channel) and how partitions/keys enforce it.
-
Multi-Region Architecture, Failover, and Disaster Recovery
-
Active-active by region with data residency.
-
Control-plane and data-plane split; inter-region replication where lawful.
-
Provider redundancy and failover strategy.
-
RTO/RPO targets and DR workflows.
-
Capacity Planning (Rough Estimates)
-
QPS, throughput, partitions, worker counts, storage footprint, cache sizing.
-
Include formulas and a worked example for tens of millions of users.
-
Monitoring and Alerting
-
SLOs/SLIs, key metrics, logs/traces, synthetic checks.
-
Alerting policies and on-call runbooks.
State key assumptions, call out trade-offs, and justify major design choices.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
-
State explicit assumptions before making sizing or architecture decisions.
-
Prioritize the functional path first, then address reliability, security, observability, and rollout.
What a Strong Answer Covers
-
A scoped requirements summary with concrete non-goals and success metrics.
-
API, data model, architecture, consistency, capacity, and operations.
-
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
-
A validation, monitoring, migration, and launch plan appropriate for the risk level.
Follow-up Questions
-
What breaks first at 10x traffic or data volume?
-
How would you degrade gracefully during dependency failures?
-
What metrics and alerts would prove the design is healthy after launch?