Design a Multi-Tenant Enterprise Messaging System

## Scenario Design an enterprise messaging system with direct messages and channels. Messages are durable, online users receive them in near real time, offline users can catch up, and users can search history. Organizations are separate tenants, and data from one organization must never be exposed to another. Support public, private, and invitation-only channels, administrator roles, read position, and online presence. Explain ordering, storage, permissions, fan-out, search, and tenant isolation. ### Constraints & Assumptions - A user belongs to one or more organizations but has an organization-scoped identity and permissions in each. - Messages in one conversation require a deterministic server order; order across conversations is unnecessary. - Channels can have hundreds or many thousands of members. - Clients reconnect and may resend a message after losing an acknowledgment. - Search results and index data are subject to the same tenant and conversation permissions as message reads. ### Clarifying Questions to Ask - What peak messages per second, tenant sizes, retention, and attachment limits apply? - Is exact presence required, and how stale may it be? - Are edits, deletion, threads, reactions, and compliance holds in scope? - What delivery guarantee and offline-notification behavior is expected? - Do large tenants require dedicated infrastructure or data residency? ```hint Put tenant identity in every authority boundary Authentication context, storage keys, cache keys, event partitions, search documents, and audit records must carry a verified tenant ID. ``` ```hint Store read progress, not one row per message read A per-user, per-conversation highest-read sequence avoids channel-size multiplied by message-count write amplification. ``` ### What a Strong Answer Covers - Tenant, user, conversation, membership, message, read-position, and idempotency data models. - A server-assigned per-conversation sequence and a durable write before real-time fan-out. - Gateway connections, reconnect cursors, deduplicated sends, offline delivery, and backpressure. - Strong server-side membership and role checks for every write, read, fan-out, and search result. - Tenant isolation choices ranging from shared rows with enforced policies to dedicated storage for selected tenants. - Read-receipt compression, presence leases, message partitioning, and large-channel fan-out trade-offs. - Permission-aware search indexing, deletion or retention handling, observability, audit, and recovery. ### Follow-up Questions 1. How do two gateway nodes agree on message order within one channel? 2. What happens when the database commits but the real-time publish fails? 3. How does a private-channel removal propagate to caches and search? 4. When should fan-out happen on write versus on read? 5. How can a shared-table design mechanically prevent a missing tenant predicate?