Peer-to-Peer File Distribution: Discovery, Chunked Transfer and Integrity Checks

You are asked to design a peer-to-peer file sharing system. Users publish files they hold, and other users download those files directly from the peers that already have copies instead of from a central file server. The interviewer wants to see how you identify and locate content, how you move large files efficiently between unreliable machines, and how you keep downloads correct while peers come and go. Only the topic of the round was reported, so the scope below is a standard framing stated as assumptions. Confirm or adjust it with the interviewer before you start designing. ### Constraints and Clarifications - Peers are ordinary end-user machines: they join and leave at any time, their upload bandwidth is usually much lower than their download bandwidth, and many of them sit behind NAT routers or firewalls. - File bytes travel peer to peer. Any central component you add should handle metadata and coordination only, never file contents. - Files can be large (several gigabytes), so a download must survive client restarts and pull from several sources at once. - Some peers are buggy or malicious and may send corrupted data. A client must detect bad data and never write it into the finished file. - Content search, user accounts and payments are out of scope unless the interviewer adds them. ### Clarifying Questions - How many concurrent peers and shared files should the design target, and how much more popular is the hottest file than the long tail? - Is a coordination service that we operate acceptable, or must the system work with no server of ours at all? - Are files immutable once published, or can a publisher release new versions under the same name? - Is every published file available to anyone who has its identifier, or do we need private shares? - Which clients matter: desktop only, or also mobile devices with metered data and battery limits? ### Part 1 — Content identity and peer discovery Explain how a file is identified and described, and how a peer that wants a file finds other peers holding all or part of it. Compare at least two discovery approaches and say which one you would start with. ```hint Name the bytes, not the file Think about what identifier lets two strangers agree they mean exactly the same content, and what a downloader must know before it fetches the first byte. ``` ```hint Who answers "who has it?" Weigh the operating and failure cost of a component you run against a lookup structure that the peers maintain among themselves. ``` #### What This Part Should Cover - A content-derived identifier and the metadata a downloader needs up front - Discovery mechanisms and the trade-off between a coordinator and a decentralized lookup - How peer lists stay fresh as peers join and leave ### Part 2 — Transferring the file Describe how a downloader actually gets the bytes from many peers: how the file is split, which pieces it asks for and from whom, how it verifies what it receives, and how it resumes after a crash or restart. ```hint Choose the unit of work Decide what unit a peer requests, verifies and advertises, and what happens when the sender disappears halfway through one. ``` ```hint Which piece next? If every downloader fetched pieces in file order, picture the swarm at the moment the original publisher goes offline. ``` #### What This Part Should Cover - Splitting the file and verifying each part against trusted hashes - Piece and peer selection, including how a brand-new peer gets started and how the download finishes - Resume and storage behavior on the downloading client ### Part 3 — Making the swarm work at scale Explain how the system keeps working with real-world peers: peers behind NAT, peers that only download and never upload, a file that suddenly becomes very popular, and a file whose last complete copy is about to disappear. Say what you would monitor. ```hint Reachability first Two peers that are both behind home routers cannot simply connect to each other. Think about what third party could help them, and what to do when even that fails. ``` ```hint Reward uploading Consider how a peer could choose whom to upload to so that contributing bandwidth pays off for the contributor. ``` #### What This Part Should Cover - Connectivity across NAT and firewalls, with a fallback - Incentives against free-riders and how load spreads when a file turns popular - Availability under churn and failure handling in the coordination layer - Observability for a system whose machines the operator does not control ### What a Strong Answer Covers - Separates the coordination plane from the peer-to-peer data plane and keeps file bytes off central servers - Treats every remote peer as untrusted, with integrity rooted in an identifier the user obtained from a trusted link - Designs for churn and asymmetric bandwidth from the start rather than as an afterthought - Justifies discovery and piece-selection choices with concrete trade-offs instead of only naming protocols - Quantifies the load on any central component and shows it stays small compared with the data peers move ### Follow-up Questions - How would you publish a new version of a large file so that peers holding the old version download only what changed? - How would you restrict a file to invited users without any central server seeing its contents? - A popular file is being poisoned: many peers advertise it but serve garbage pieces. How do you detect and contain them? - How would the design change for mobile clients on metered connections that cannot keep uploading in the background?