Notification And Rate Limiter Service Design

What's being tested

This design tests whether you can decompose a notification platform and a distributed rate limiter into clear APIs, storage models, asynchronous workflows, and failure boundaries. The interviewer is probing your command of consistency, idempotency, backpressure, retries, and multi-tenant isolation—not merely whether you can name `Redis` and `Kafka`. For Affirm-like financial workflows, correctness matters: duplicate, delayed, or missing notifications can create user confusion, while an ineffective limiter can overload downstream providers or allow abusive traffic.

Core knowledge

• Requirements clarification should establish notification channels (`email`, `SMS`, push), delivery guarantees, expected throughput, latency targets, retention, tenant/user quotas, and whether ordering is required. Separate business-critical notifications from best-effort marketing or informational messages.

• Use an API gateway or service endpoint for synchronous validation, then persist an immutable notification request before returning success. A durable outbox pattern prevents the classic failure where the database commit succeeds but publishing the asynchronous event fails.

• Model the workflow as a durable state machine: `accepted` → `queued` → `sending` → `delivered`, with terminal states such as `failed`, `expired`, or `suppressed`. Store provider message IDs, attempt counts, timestamps, and failure classifications for reconciliation and debugging.

• Use a durable queue such as `Kafka`, `SQS`, or `Pub/Sub` between intake and delivery workers. Partitioning by recipient or notification key can preserve per-user ordering, but global ordering is expensive and usually unnecessary.

• Delivery is generally at least once, so consumers must be idempotent. Require a caller-supplied idempotency key or derive a stable event ID; enforce uniqueness in `Postgres` or a distributed key store before sending. Exactly-once external delivery is usually impossible without provider cooperation.

• Implement retries with exponential backoff and jitter, for example $d_n=\min(d_{\max},d_0 2^n)+U(0,j)$. Retry transient network errors, provider `429`, and selected `5xx`; do not blindly retry invalid addresses, revoked tokens, or permanent provider errors.

• A token bucket limiter allows bursts while enforcing an average rate: tokens refill at rate $r$ up to capacity $B$, and each request consumes one token. A request is accepted when available tokens $\geq 1$; otherwise return, delay, or queue it.

• A sliding-window counter is simpler but less precise at boundaries; fixed windows can permit nearly $2L$ requests around a reset. For distributed enforcement, store bucket state atomically in `Redis` using Lua or a server-side transaction, and define clock and TTL behavior explicitly.

• Choose limiter scope deliberately: per API key, merchant, user, destination, provider, channel, and globally. Apply hierarchical limits from narrowest to broadest; an abusive tenant should not consume the entire global SMS capacity.

• Backpressure prevents overload when downstream providers slow down. Bound queue depth and worker concurrency, expose saturation metrics, and decide whether excess work is rejected, delayed, expired, or routed to a fallback channel. Unbounded queues merely convert overload into latency and eventual memory failure.

• Provider adapters should normalize responses into an internal contract containing `accepted`, `retryable`, `permanent_failure`, and `unknown` outcomes. An `unknown` result after a timeout is dangerous: retrying may duplicate delivery, so reconcile using the provider’s idempotency key or status API when available.

• Define observability around operational behavior: acceptance rate, queue age, delivery success rate, retry volume, provider `429` rate, limiter rejection rate, and `p99` delivery latency. Correlate every request through a trace ID, notification ID, and provider ID without logging message bodies or sensitive financial data.

Worked example — Design a Notification and Rate Limiter Service

A strong candidate first clarifies whether the goal is an API for submitting notifications, a delivery system, a standalone limiter, or all three, then asks about channels, peak traffic, delivery guarantees, ordering, and tenant-specific quotas. They might declare an assumption of 100,000 requests per second at intake, at-least-once processing, per-user ordering, and no guarantee of exactly-once external delivery. The answer can then be organized around four pillars: an idempotent intake API and durable outbox, a partitioned queue with worker state machines, a provider-adapter and retry strategy, and a distributed hierarchical limiter. They should explicitly choose a token bucket in `Redis` with an atomic Lua script for fast admission, while acknowledging that durable queue limits still need enforcement outside the cache. They should flag the tradeoff between strict ordering and throughput: partitioning by user preserves local order but can create hot partitions, so ordering should be limited to notification classes that require it. They should explain that a provider timeout produces an ambiguous result and requires reconciliation rather than unconditional retry. They should close by saying that, with more time, they would detail multi-region behavior, provider failover, key expiration, quota configuration rollout, and a failure-injection test plan.

A second angle

Consider a notification service where the dominant constraint is a provider quota rather than API traffic: `SMS` permits only 500 messages per second, while email has much higher capacity. The limiter must then sit close to each provider adapter and account for channel, destination country, tenant, and provider-specific quotas. A global intake limiter still protects the service, but queue scheduling and weighted fairness become more important than simple request rejection. The design should also address expiration, because a delayed one-time code may be useless even if it eventually sends.

Common pitfalls

Pitfall: Treating a `Redis` increment as sufficient for distributed rate limiting ignores race conditions, clock behavior, and multi-key hierarchical limits. Use an atomic token-bucket operation, define TTLs, and explain what happens when the limiter store is unavailable.

A tempting answer promises exactly-once notification delivery by combining a database transaction with a queue. That only gives durable intent; the external provider call can still time out after accepting the message. Say “at least once internally, idempotent where supported, and reconciled for ambiguous outcomes” instead.

Candidates often enumerate components without establishing requirements or failure semantics. Start with scale, latency, ordering, quotas, and delivery guarantees, then connect each choice to those assumptions; this demonstrates engineering judgment rather than architecture-name memorization.

Connections

The interviewer may pivot to distributed locks, outbox and inbox patterns, fair scheduling, dead-letter queues, or multi-region consistency. Be ready to compare `Redis`-based admission with gateway-native limiters and to explain how circuit breakers interact with retries and backpressure.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s treatment of streams, delivery semantics, partitioning, and consistency.

• The Tail at Scale — why `p99` latency, hedging, and failure amplification matter in distributed services.

• Amazon Builders’ Library: Timeouts, retries, and backoff with jitter — practical guidance for avoiding retry storms and overload cascades.