Multi-Tenant Architecture And Tenant Isolation

What's being tested

These designs test whether you can build multi-tenant systems that scale while preventing one customer from reading, mutating, or exhausting another customer's resources. The interviewer is probing tenant-aware data modeling, authorization boundaries, asynchronous processing, consistency, failure recovery, and operational isolation. Salesforce operates large shared platforms, so a strong answer must balance efficiency of pooled infrastructure against hard security boundaries, predictable per-tenant performance, and explainable failure behavior.

Core knowledge

• Tenant context propagation must begin at authentication and travel through every request, job, query, cache key, topic, and storage path; never trust a client-supplied tenantId without deriving it from verified identity and authorization claims.

• Isolation models range from shared tables with tenant_id, to separate schemas, to separate databases or accounts. Shared tables maximize utilization; dedicated databases improve blast-radius and compliance isolation but increase provisioning, migration, and operational cost.

• Enforce isolation redundantly: application authorization, repository APIs requiring tenant context, database row-level security such as `Postgres` RLS, tenant-qualified indexes, and tests that attempt cross-tenant reads. Defense in depth catches missing predicates and confused-deputy bugs.

• For shared data, use composite indexes such as (tenant_id, channel_id, created_at) and consider partitioning by tenant or time. A query filtering only channel_id is both slower and a potential isolation defect; make tenant scope structurally difficult to omit.

• Noisy-neighbor control requires per-tenant quotas, rate limits, bounded worker concurrency, queue fairness, and circuit breakers. A weighted fair scheduler or per-tenant token bucket prevents a large customer from consuming all `Kafka` partitions, storage bandwidth, or websocket capacity.

• In messaging, separate durable writes from real-time delivery: persist an event with a monotonic sequence, then publish asynchronously. Clients reconnect using a cursor or read position; delivery is typically at-least-once, so consumers need idempotency and deduplication.

• Ordering must be scoped explicitly: global ordering is expensive and unnecessary; channel ordering can use a partition key such as (tenant_id, channel_id). If multiple writers exist, use database sequence numbers, logical clocks, or a single ordered partition.

• Fan-out depends on recipient count. Store once and resolve membership on read for large channels, or create recipient inbox entries for small channels requiring fast unread counts; hybrid fan-out avoids multiplying storage by millions of users.

• Large-file replication should use multipart, checksummed, resumable transfers with a manifest containing tenant, object version, destination, and status. Upload to a temporary key, verify the complete checksum, then atomically mark the destination available.

• Idempotency is mandatory for retries: identify work by (tenant_id, object_id, version, destination) and make repeated copy, allocation, or delivery operations converge to one result. Exponential backoff with jitter handles transient failures; dead-letter queues retain unrecoverable work.

• Model resource allocation as an explicit state machine, for example AVAILABLE → HELD → ALLOCATED → RELEASING → AVAILABLE, and reject illegal transitions. Store an owner, lease expiry, version, and timestamps so recovery can distinguish expiration from deliberate release.

• Optimistic locking prevents lost updates using UPDATE ... WHERE id = ? AND version = ?, requiring exactly one affected row before incrementing version. It works well for contention that is usually low; high contention may require sharding, queues, or pessimistic locking.

• Permissions are separate from tenant identity: users may belong to multiple tenants, channels can have roles, and administrators may have scoped cross-tenant support access. Check authorization at every operation, not only at login or API routing.

• Measure isolation as well as correctness: track per-tenant `p50`/`p99` latency, queue age, error rate, throttling, replication lag, and storage growth. Alert on cross-tenant access attempts and unusual tenant-level consumption, while keeping logs tenant-scoped and privacy-safe.

Worked example: Design a Multi-Tenant Enterprise Messaging System

A strong candidate first clarifies tenant count, peak messages per second, channel-size distribution, retention and search requirements, delivery guarantees, compliance constraints, and whether a user can belong to multiple tenants. They then state assumptions—for example, at-least-once delivery, per-channel ordering, durable history, and reconnectable clients using cursors. The answer can be organized around four pillars: tenant-safe identity and permissions, durable message storage, asynchronous fan-out and real-time delivery, and search/presence/read-position services. Every event and cache key should carry verified tenant context, while channel membership checks prevent both cross-tenant access and unauthorized same-tenant reads. A specific tradeoff is whether to fan out messages on write or read: write fan-out gives fast inbox reads but can explode for huge channels, so a hybrid threshold is more robust. The candidate should explain backpressure, reconnect semantics, idempotent message IDs, and how a failed websocket gateway catches up from durable storage. They should close by saying that, with more time, they would detail retention deletion, encryption and key isolation, multi-region failover, and load-testing methodology.

A second angle

For Design a tenant-isolated system to replicate large recordings to per-customer data centers, the central issue shifts from low-latency fan-out to durable, bandwidth-heavy workflows and destination-specific isolation. The design should emphasize manifests, multipart transfer, checksums, resumability, per-tenant scheduling, and explicit lifecycle states rather than websocket delivery. A shared transfer service can be efficient, but quotas and weighted scheduling must prevent one tenant's recordings from monopolizing network or disk. The same tenant context and idempotency rules apply at every queue, object path, retry, and destination.

Common pitfalls

Pitfall: Treating tenant_id as merely a filter added by convention is an analytical mistake. A missing predicate, unscoped cache key, or background job can become a data leak; stronger answers enforce scope through APIs, database policies, indexes, and adversarial tests.

A communication mistake is jumping directly into `Kafka`, sharding, or multi-region replication before clarifying traffic, retention, ordering, and isolation requirements. State assumptions first, draw the trust boundary, and explain which guarantees are required versus optional.

Pitfall: Saying “use a transaction” or “make it strongly consistent” without defining the boundary is too shallow. Explain which state transition, message publication, or copy completion is atomic, then address retries, partial failure, duplicate work, and recovery.

Connections

The interviewer may pivot to authorization and RBAC, database sharding and partitioning, idempotent distributed workflows, or queue fairness and backpressure. Be ready to connect tenant isolation to encryption-key scope, disaster recovery, observability, and multi-region consistency.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann’s treatment of replication, partitioning, consistency, streams, and failure modes.

• Amazon Builders’ Library: Timeouts, retries, and backoff with jitter — practical guidance for reliable asynchronous distributed systems.

• Google Zanzibar: A Global Authorization System — foundational model for scalable relationship-based authorization.