System Design: Production-Ready File Deduplication Service

Context

Design a multi-tenant cloud service that stores files and achieves space savings via deduplication. The service must handle large scale (billions of files, petabytes of data), support high availability across regions, and provide operationally safe mechanisms for change management.

Assume:

• Files are immutable once written (new versions create new files/manifests).
• Deduplication happens at the chunk level using content-defined chunking.
• Storage is an object store; metadata and indexes use scalable data stores.

Requirements

Outline the following:

1. Architecture
   • Ingest, chunking, indexing, storage, and metadata layers.
   • How manifests map files to chunks.
2. APIs
   • Read/write, streaming/multipart, idempotency, and admin/maintenance endpoints.
3. Read/Write Workflows
   • End-to-end flows, including parallelism and error handling.
4. Consistency and Safety
   • Consistency model, idempotency strategies, fault isolation, failure recovery, and disaster recovery (RPO/RTO).
5. Maintenance Operations
   • Backfills, compaction/garbage collection, index sharding and rebalancing, and safe rollout/rollback and schema/version migrations.
6. Operations and SRE
   • Monitoring, alerting, SLOs; capacity planning and cost controls (compute, storage, network).
7. Privacy and Compliance
   • Encryption, access control, GDPR/data deletion, residency, auditing.
8. Minimizing Production Impact
   • Rate limiting, backpressure, priority queues, and other isolation mechanisms.