Determine identical files ignoring metadata

##### Question How would you determine whether two files contain identical bytes while ignoring all filesystem metadata? Propose and justify a hashing-based approach, then extend it to large-scale, cross-machine deduplication. Address the following: 1. **Algorithm choice and rationale.** Cryptographic versus non-cryptographic hashing (e.g., SHA-256 vs MD5 vs BLAKE3 vs xxHash); why you would or would not use each, and which is authoritative. 2. **Multi-stage verification.** A pipeline that combines a size check, an optional fast prefilter hash, an authoritative cryptographic hash, and an optional byte-by-byte confirmation — and when each stage runs. 3. **Collision handling and false-positive mitigation.** Birthday-bound reasoning, when collisions matter, and how to defend against adversarial (chosen-prefix) inputs versus non-adversarial data. 4. **Metadata to check before hashing.** What attributes (size, mtime/ctime, inode/device id, file type) you would use to prefilter, detect hard links, and cache digests to avoid rehashing unchanged files. 5. **Efficient streaming for very large files.** Chunked/streaming reads, buffer sizing, intra-file parallelism (e.g., Merkle trees / BLAKE3), sparse-file handling, and TOCTOU guardrails for files that change mid-read. 6. **Large-scale deduplication.** Whole-file vs fixed-size vs content-defined chunking (CDC); persistent hash index design; and how a dedicated service would compute, store, and serve comparisons reliably across directories and machines. 7. **Performance trade-offs.** Where the bottleneck lies (I/O vs CPU vs memory) and how your design responds to it.