Design an object store with deduplication
Company: Snowflake
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
## System Design Prompt
Design a simplified cloud object storage service (similar to an object store) that allows users to upload and download files.
The key focus is **reducing cost by avoiding storing duplicate files** (deduplication).
### Core Requirements
- Upload a file (binary blob) and store it durably.
- Download a previously uploaded file by an identifier.
- Support basic metadata (filename, content-type, size, upload time, owner).
- Avoid saving duplicates: if multiple users upload the exact same content, store it only once.
### Non-Goals (to keep it simpler)
- No complex lifecycle rules, tiering, or CDN features required.
- Permissions can be simplified (assume authenticated users, simple ACL).
### Scale / Assumptions (you may choose reasonable numbers)
- Large number of objects; objects can be large (up to multiple GB).
- High read/write throughput.
- Must be resilient to failures.
### What to Cover
- APIs
- High-level architecture
- Data model and storage layout
- Deduplication approach (how to detect duplicates, how to reference shared content)
- Consistency and correctness concerns (races, partial uploads)
- Deletion / garbage collection (when shared content can be removed)
- Operational considerations (monitoring, cost trade-offs)
Overview: This question evaluates a candidate's ability to design scalable, durable object storage systems with deduplication, touching on storage architecture, data modeling, metadata management, consistency, and garbage collection.
Community answers
Answer by siddhantaws
S3 like storage
file size limit min to max - 100byte to 1TB
-What I will design end end to api to store
Durability number 99.11th 9
Deduplication
Rack level failure handling
Function requiment
UPLOAD,GET,DELETE
Deduplication
file versioning
avoid race condition during garbage coletion
Non functional requirement
Durability
Avalibility>consistenct 99.999 avalibility 5min
-p95 query time < 500ms
Avoid
Entity
bucket { id , name , visibility}
Objects { object_id , status, key , upload_by , }
Objects_chunks {id, object_id , chink_index , chunk_hash}
chunks { chunk_hash , refer_count , is_deleted, updated_at}
Back of envelope calculation
10B Object eeach 1MB
Total 10PB , 30PB
100MDAU , 2 Object each size 1MB
2TB per day +
800MB , 8GB Mbps
Rest API
POST /api/v1/{bucket}/{key} {
content_length : 3123132312
content_type "",
encoding :
{
"algo" : "sha-1",
"value" : "4234dfsdfsdf"
},
{
"algo" : "md5",
"value" : "4234dfsdfsdf"
}
"chunk_size" : 64MB
}
201 api/v1/upload/{upload_id} {
chunks : [ {"chunk_id" : 1 , "signed_url" : "ama" , st_range : 0,"ed_range" : "", expire_at} ]
}
302 api/v1/upload/{upload_id} {
}
201
api/v1/upload/{upload_id} {"chunk_idx" "1" , etag : }
api/v1/upload/{upload_id}/complete
GET api/v1/{upload}/
201 api/v1/{upload}/
{
"chunk" : 1 , "signed_url" : "", expire_at:""
}
Zookeeper
Monitor the health of storage node and give callback to monitoring service so that re-replication will happen
Replication will happen in priority order so that higher priority means lower im