Design Duplicate File Detection
Company: OpenAI
Role: Machine Learning Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Design a system to find duplicate files.
Start with a single-machine version: given a large directory tree, identify groups of files that have identical contents even if their file names and paths are different.
Then discuss how you would optimize and extend the solution to a large distributed storage environment with many machines and a very large number of files.
Your design should address:
- How to avoid unnecessary full file reads
- How to compare files efficiently and safely
- How to manage CPU, memory, and disk I/O
- How to support incremental rescans when files are added or modified
- How to partition work across machines
- How to aggregate and verify duplicate groups in a distributed setting
Quick Answer: This question evaluates system design and storage engineering skills, including scalable file-processing, efficient I/O and resource management, correctness in content-based duplicate detection, and distributed aggregation.