Slack-Like Real-Time Messaging

What's being tested

Interviewers probe the candidate’s ability to design a real-time messaging system that balances low-latency delivery, durability, and multi-tenant isolation at scale. Expect to demonstrate distributed-systems patterns (pub/sub, partitioning, backpressure), data modeling for channels/threads, and operational tradeoffs (ordering, retries, and consistency). They're checking for practical decisions—protocols, storage choices, and how you reason about scale, failure modes, and developer-facing APIs.

Core knowledge

• Transport layer: choose between `WebSocket`, HTTP long-polling or HTTP/2/gRPC streams; `WebSocket` gives bi-directional low-latency channels but requires sticky routing or connection multiplexing for scale.

• Pub/Sub vs direct: a publish/subscribe broker (e.g., `Kafka`, `NATS`) decouples producers/consumers; brokers trade off ordering and latency depending on partitioning and replication.

• Partitioning & sharding: shard by workspace or by channel id to keep ordering and reduce fanout; partitions count ≈ required throughput / per-partition throughput, adjust for hot channels.

• Fanout strategies: push-based fanout (server pushes to every connected client) is lowest-latency but expensive for large rooms; pull-based or lazy delivery (store-and-notify) reduces CPU at cost of latency and complexity.

• Ordering semantics: pick application-level guarantees: per-channel FIFO via sequence numbers, or causal ordering with vector clocks; global ordering is expensive and rarely necessary.

• Durability model: combine a fast ephemeral store (`Redis`) for presence and recent messages with durable storage (`Postgres` or append-only `Kafka` + cold store) for history and compliance.

• Delivery guarantees: implement at-least-once with deduplication tokens or exactly-once semantics via idempotent writes where required; prefer idempotent consumers rather than distributed transactions.

• Presence & typing: maintain presence as ephemeral keys in `Redis` with TTLs and heartbeat; propagate presence events over the same pub/sub channel to avoid extra endpoints.

• Security & multi-tenancy: tenant isolation via scoped topics/partitions, per-tenant encryption keys, `OAuth` access tokens, and tenant-aware rate limiting; audit logs stored in immutable append-only stores for compliance.

• Backpressure & flow control: implement per-connection buffers, token-bucket rate limiting, and server-side drop/slow-path (e.g., send only summaries) when clients can't keep up.

• Observability & SLOs: monitor `p99` and `p50` end-to-end latency, message loss rate, and consumer lag (`Kafka` offsets) and expose tenant-level SLIs and throttles.

• Scaling calculations: estimate RPS × avg message size × average connected clients to dimension network and broker throughput; use $\text{bandwidth} = RPS \times \text{avg\_msg\_size} \times \text{avg\_fanout}$.

Worked example — "Design a Slack-Like Messaging System"

First 30 seconds: clarify requirements — expected number of concurrent users, max channel size, mobile vs web clients, retention/compliance, ordering and durability guarantees, and whether presence/typing is required. Assumptions: support millions of users, per-channel FIFO ordering, message history persisted for 30 days, primary client is `WebSocket`-enabled web/mobile.

Skeleton answer pillars: (1) API & client protocol — auth (token), `WebSocket` for realtime events, REST for history; (2) Realtime layer — stateless frontends with sticky connection routing to a pool of sharding gateways that forward publishes to a message broker; (3) Broker & storage — partitioned append-log (`Kafka`) for durability plus `Redis` for recent messages and presence; (4) Fanout & delivery — brokers push to gateways which manage per-connection buffers and ack/dedup logic; (5) Operational — monitoring, rate-limits, and tenant isolation.

Key tradeoff: choose per-channel partitioning to keep FIFO ordering and minimize coordination, but accept that very large channels (e.g., #general) become hot partitions and need fanout workarounds (replicated fanout nodes or batching). Explain mitigation: backpressure + summarized notifications + archival for heavy channels.

Close: state next steps — prototype with `Kafka` + `Redis`, stress-test hot channels, design migration for sharding splits, and if more time, add end-to-end encryption and per-tenant retention policies.

A second angle — "Design a multi-tenant Slack-like messenger"

Here the framing emphasizes tenant isolation and per-tenant policies. Start by clarifying tenant scale distribution (many small tenants vs few large ones), legal/regulatory requirements, and allowed cross-tenant features (shared apps). The core architecture is similar, but you must add tenant-aware routing (topics prefixed by tenant-id), per-tenant quotas (rate limits, storage caps), and logical isolation in observability/alerts. For noisy neighbors, use per-tenant queues or rate-limited gateways and consider soft vs hard multi-tenant isolation: soft (shared infra with quotas) is cheaper; hard (separate clusters or VPCs) required for high compliance tenants.

Common pitfalls

Pitfall: conflating transport with persistence. Candidates often assume `WebSocket` implies persistence — you still need durable storage and a replay mechanism; design the retention and replay API explicitly.

Pitfall: promising global ordering. Saying "messages are globally ordered" is tempting but impractical; prefer per-channel ordering and explain why global ordering requires expensive consensus and hurts latency.

Pitfall: ignoring hot channels and backpressure. A common mistake is to assume equal load distribution; instead show how you'll detect hot rooms, shard/split them, and provide degraded UX (summaries) under overload.

Connections

This topic commonly leads to adjacent systems: notification/push services (APNs/FCM) for mobile delivery and search/indexing (message search with `Elasticsearch` or `Opensearch`). Interviewers may also pivot to consistency models, distributed consensus (raft), or designing an audit/logging pipeline for compliance.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann — principled tradeoffs for logs, replication, and partitioning.

• Slack Engineering Blog — How Slack Scales (various posts) — practical notes on real-time design and scaling challenges.