Find Duplicate Files with a Hand-Written Directory Traversal
Quick Overview
Build a duplicate-file finder without recursive walking helpers. This exercise covers iterative directory traversal, bounded-memory content comparison, collision-safe verification, deterministic results, symlink and hard-link policy, I/O failures, and races when files change during a scan.
Find Duplicate Files with a Hand-Written Directory Traversal
Company: Snapchat
Role: Software Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Technical Screen
# Find Duplicate Files with a Hand-Written Directory Traversal
Implement a duplicate-file finder starting from a root directory, but do not use a recursive walking helper such as `Files.walk` or `os.walk`. You may use only low-level operations to list one directory, inspect one directory entry, and stream one file's bytes.
Return groups of paths whose file contents are byte-for-byte identical. Include only groups containing at least two files. Explain how you keep memory bounded for large files and how you protect correctness from hash collisions.
### Constraints & Assumptions
- Directory depth may exceed the safe recursion depth.
- Files may be empty, very large, or unreadable.
- Directory enumeration order is unspecified.
- By default, do not follow symbolic links; state how the design changes if links must be followed.
- The result order must be deterministic for testing.
### Clarifying Questions to Ask
- Should hard links to the same underlying file appear as duplicates or as one object?
- Should permission and transient I/O errors fail the scan or be returned separately?
- Is a cryptographic digest sufficient, or is byte-for-byte verification required?
- Can files change while the scan is running?
### What a Strong Answer Covers
- An explicit stack or queue for manual traversal
- Filtering by file size before hashing
- Streaming digests and final equality verification
- Cycle, symlink, error, and concurrent-modification policies
- Deterministic output and clear complexity bounds
### Follow-up Questions
- How would you reuse work across repeated scans?
- What race occurs if a file changes between metadata lookup and hashing?
- How could the scanner exploit parallel I/O without overwhelming storage?
- Would you delete duplicate files automatically based on this result?
Quick Answer: Build a duplicate-file finder without recursive walking helpers. This exercise covers iterative directory traversal, bounded-memory content comparison, collision-safe verification, deterministic results, symlink and hard-link policy, I/O failures, and races when files change during a scan.
Find Duplicate Files with a Hand-Written Directory Traversal
Implement a duplicate-file finder starting from a root directory, but do not use a recursive walking helper such as Files.walk or os.walk. You may use only low-level operations to list one directory, inspect one directory entry, and stream one file's bytes.
Return groups of paths whose file contents are byte-for-byte identical. Include only groups containing at least two files. Explain how you keep memory bounded for large files and how you protect correctness from hash collisions.
Constraints & Assumptions
Directory depth may exceed the safe recursion depth.
Files may be empty, very large, or unreadable.
Directory enumeration order is unspecified.
By default, do not follow symbolic links; state how the design changes if links must be followed.
The result order must be deterministic for testing.
Clarifying Questions to Ask Guidance
Should hard links to the same underlying file appear as duplicates or as one object?
Should permission and transient I/O errors fail the scan or be returned separately?
Is a cryptographic digest sufficient, or is byte-for-byte verification required?
Can files change while the scan is running?
What a Strong Answer Covers Guidance
An explicit stack or queue for manual traversal
Filtering by file size before hashing
Streaming digests and final equality verification
Cycle, symlink, error, and concurrent-modification policies
Deterministic output and clear complexity bounds
Follow-up Questions Guidance
How would you reuse work across repeated scans?
What race occurs if a file changes between metadata lookup and hashing?
How could the scanner exploit parallel I/O without overwhelming storage?
Would you delete duplicate files automatically based on this result?