Design file deduplication at scale
Company: Anthropic
Role: Software Engineer
Category: Coding & Algorithms
Difficulty: medium
Interview Round: Onsite
Design an algorithm to identify duplicate files in a large directory tree. You are given an iterator over files providing (path, size) and a function read_chunks(path) -> Iterator[bytes]. Requirements: minimize I/O by comparing sizes and using rolling or cryptographic hashes; handle hash collisions safely; support datasets that do not fit in memory; and output groups of paths that are byte-for-byte identical. Explain time/space trade-offs and how you would parallelize the solution.
Overview: This question evaluates understanding of file-system algorithms, hashing and deduplication techniques, scalable I/O and memory management, and correctness considerations for hash collisions, and it belongs to the Coding & Algorithms domain.
Read the full Anthropic Software Engineer interview experience this question came from