Real-Time Notifications And WebSockets For Banking

What's being tested

Interviewers are probing your ability to design a low-latency, reliable push-notification system for banking use cases that respects security, scalability, and operational constraints. They want to see system decomposition into connection management, message routing/fanout, delivery semantics, failure/reconnect strategies, and measurable SLOs (e.g., `p99` latency, delivery guarantees). Expect questions that require tradeoffs (exactly-once vs at-least-once, sticky sessions vs stateless gateways) and concrete capacity/operational considerations.

Core knowledge

WebSocket vs SSE vs long polling: WebSockets provide bidirectional, low-latency channels suitable for interactive banking events; `SSE` is unidirectional (server→client) and simpler; long polling is fallback for legacy environments.
WSS/TLS and auth: Always use TLS (WSS). Authenticate with short-lived JWT or ephemeral tokens issued at login; include server-side authorization checks per message to prevent horizontal privilege escalation.
Connection capacity math: total memory ≈ N_conn * mem_per_conn + process_overhead; example: if mem_per_conn=2KB and N_conn=50k → ~100MB just for sockets. Consider file-descriptor limits (`ulimit -n`) and kernel tuning.
Load balancing & sticky sessions: sticky sessions route the same user to the same gateway; stateless routing requires a message broker (e.g., `Kafka`, `Redis Streams`) + fanout layer to avoid session affinity tradeoffs.
Fanout patterns: use topic-per-tenant or topic-per-event-type with consumer groups for backend processing, then fanout to connection gateways. For high-cardinality per-user channels, store connection -> node mapping in a fast KV (e.g., `Redis`) for direct routing.
Delivery semantics & sequencing: use sequence numbers and client ACKs for per-channel ordering; server persists sequence checkpoints when at-least-once delivery is required. Exactly-once generally requires idempotent handlers and dedup keys.
Backpressure & batching: implement per-connection send-queues with bounded capacity, drop/merge non-critical updates, and expose queue-length metrics. Batch small messages to reduce `syscall` overhead and improve throughput.
Reconnect and resume: support reconnection with exponential backoff + jitter; allow resume tokens (session ID + last seen sequence) so server can send missed messages from a persistent queue.
Monitoring & SLOs: capture `connections`, `messages/sec`, `p50/p95/p99` latencies, `reconnects`, `auth_failures`, and error rates; set SLOs for delivery latency (e.g., 99% < 300ms for real-time notifications).
Operational: use a WebSocket gateway (`Envoy`, `NGINX`, managed `API Gateway`) fronting app nodes; horizontally scale gateways and shard connections; tune OS TCP ephemeral ports, keepalive, and memory limits.
Testing: load-test with `k6`, `wrk`, or `gatling` websocket plugins; test failover (kill nodes), token expiration, replays, and high fanout scenarios; simulate mobile network flakiness for reconnect behavior.

Worked example — "Design a real-time notification system for banking customers using WebSockets"

First 30 seconds: clarify requirements — targeted delivery (per-user vs broadcast), delivery guarantees (at-least-once vs exactly-once), latency SLOs (`p99` target), expected concurrent connections and peak messages/sec, mobile vs web clients. Assumptions: support 500k concurrent connections, at-least-once delivery acceptable, small message sizes (~200 bytes), strict auth required.

Skeleton answer pillars: (1) Connection gateway layer (TLS termination, auth, heartbeat, backpressure) using `Envoy` or managed gateway. (2) Connection registry in `Redis` mapping user→gateway node for direct routing. (3) Message bus (`Kafka` or `Redis Streams`) for durable ingestion and fanout to gateways. (4) Persistence/resume: store recent N messages per user in a compact persistent store for reconnection resume. Flag tradeoff: durable per-user queues increase storage and complexity; for high-cardinality users, favor short retention + client resume window. Close: mention rate limiting per-user, testing plan (load test, chaos), and if more time, add support for multi-device per-user conflict resolution and per-message encryption-at-rest.

A second angle — "How to ensure at-least-once delivery and idempotent notification handling"

Frame it as protocol plus server-side design: require client ACKs with sequence numbers; on ingest, persist message to durable `Kafka` topic; brokers deliver to gateway which retries until ACK or retention window expires. Use idempotency keys on server-side handlers and include message IDs so clients can deduplicate. Discuss tradeoffs: retries increase duplicate risk and cost; idempotency storage TTL must balance memory vs duplicate exposure. For banking, also add an audit trail for delivered messages (compliance requirement).

Common pitfalls

Pitfall: treating WebSocket servers as stateless without a routing layer. If you rely on DNS/LB for distribution without a connection registry or broker, you’ll end up with undeliverable messages or heavy sticky-session requirements.

Pitfall: assuming exactly-once delivery for free. Promising exactly-once across distributed gateways without idempotency, sequence checks, and durable storage is incorrect; instead design for at-least-once plus idempotent consumers.

Pitfall: neglecting mobile network realities. Mobile clients frequently disconnect; failing to implement resume tokens, jittered exponential backoff, and compact replay windows leads to poor UX and heavy server-side reprocessing.

Connections

Interviewers may pivot to adjacent areas: data retention/auditing for compliance, push-notification fallbacks (`APNs`/`FCM`) for mobile when sockets are unavailable, or rate-limiting/fraud-detection logic applied to outgoing notifications.