Design a tenant-isolated system to replicate large recordings to per-customer data centers

You are in a senior system design round. Design a system that moves large files, such as call recordings, from the data center where they are produced to one or more data centers chosen per customer. Each customer is a tenant of the platform, and a tenant's files must only ever reach that tenant's own destinations. The system must not lose files, and the business needs to know, for every file, whether it has safely arrived everywhere it should. Build the design up in stages: start with the simplest version that works in a single region, add asynchronous replication with queues, then address failures, retries and consistency at scale. ### Constraints and Clarifications - File sizes, daily volume and latency targets are not given; ask for them and keep your estimates parametric until you have them. - Assume files are immutable once the recording finishes. - Assume each tenant's set of destination data centers is configuration held by the platform and can change over time. - Whether the source copy is deleted after transfer is an open policy question; ask rather than assume. ### Clarifying Questions - What is the size distribution of a file, and how many files are produced per day at peak? - How soon after a recording ends must it be available at each destination? - Is this a copy (the source keeps the file) or a move (the source deletes it once every destination confirms)? - Do destinations belong to the platform in other regions, or to the customer, and how are they reached? - Must a file become readable at a destination only once it is complete and verified? - Are there residency rules about where a tenant's file may be stored, even temporarily? ### Part 1 — The simplest working version Describe a first design for one source and one destination per tenant: components, data flow, and where it breaks. ```hint Start from the moment a recording ends List what has to happen between "file closed at the source" and "file safely readable at the destination", and notice which steps are slow or fragile. ``` #### What This Part Should Cover - An end-to-end flow with the components named - Metadata about each file and its transfer status - The limits of a synchronous approach for large files ### Part 2 — Asynchronous fan-out to many destinations Evolve the design so each file is replicated to every destination configured for its tenant, using queues and workers. Explain how transfer work is created, tracked and scheduled, and how large files are moved efficiently. ```hint One file, many deliveries Think about the unit of work the queue carries, and how one slow destination affects everyone else. ``` #### What This Part Should Cover - How file events become per-destination transfer tasks without being lost - Chunked, resumable transfer of large files with integrity checks - Isolation of slow or unavailable destinations from healthy ones - A data model and internal APIs for tracking replication status ### Part 3 — Failures, retries and consistency Workers crash mid-transfer, destinations go down for hours, events are delivered twice, and bytes get corrupted. Show how the system still guarantees that every file eventually arrives intact at every destination, without duplicates and without exposing a half-written file. ```hint What does "done" mean? Define the exact condition under which a transfer is marked complete, and how you would discover a file whose transfer silently never started. ``` #### What This Part Should Cover - Retry, lease and dead-letter behavior for transfer tasks - Idempotent writes and atomic visibility at the destination - Verification and reconciliation that catch lost work - Behavior during long destination outages ### Part 4 — Tenant isolation and scale Show how the design guarantees that a file can never be written to another tenant's destination, and how it stays fair and fast when one tenant produces a large backlog or the number of tenants grows. ```hint Assume a bug in the worker Ask what would stop a worker that picked the wrong destination from actually writing there. ``` #### What This Part Should Cover - Credential, path and encryption-key scoping per tenant - Fair scheduling so one tenant cannot starve others - Horizontal scaling and the real bottleneck, cross-data-center bandwidth ### What a Strong Answer Covers - A clear progression from simple to robust, with the reason for each step - Requirements and parametric estimates that drive capacity decisions - Effectively-once delivery built from at-least-once messaging and idempotent writes - Isolation enforced by infrastructure (scoped credentials, per-tenant keys), not only by code paths - Observability: replication lag, failure rates and reconciliation results ### Follow-up Questions - A tenant adds a new destination data center. How do you backfill its existing files without disturbing live traffic? - A tenant removes a destination or requests deletion of a recording. How does that propagate? - When would you rely on the storage provider's built-in cross-region replication instead of building this? - How would you prove to an auditor that a given recording never left its permitted locations?