Distribute 500 GB of Model Weights to 1,000 GPU Workers: Lower Bound and Design

A newly trained model has 500 GB of weights. The weights must be copied to every machine in a fleet of GPU workers, somewhere between 100 and 1,000 machines, before those workers can serve the new model. Each machine, including the machine that initially holds the weights, has a network interface of 10 Gbps, shared between upload and download. When asked about the network, the interviewer said to assume no particular topology and that links can be used at their full bandwidth. Start by arguing for a lower bound on how long the distribution must take, then design a system that gets close to that bound, evolving it from a simple baseline. ### Clarifying Questions - Does "10 Gbps shared between upload and download" mean the sum of a machine's upload and download rates is at most 10 Gbps, or that each direction gets 10 Gbps? - Is the source a single machine, or an object store that can serve many machines in parallel? - Does every worker need all 500 GB, or only the shard of the model that its GPUs will hold? - How often are new models pushed, and must the old model keep serving while the new one downloads? - How often do workers fail or join during a distribution? - Are workers grouped into racks where traffic inside a rack is cheaper than traffic across racks? ### Part 1 — The lower bound Derive the minimum possible time to get the full 500 GB onto all workers, for 100 and for 1,000 workers, under the stated assumptions. Show where each limit comes from, and how the answer depends on the interpretation of "shared" bandwidth. ```hint Count the bits both ways Every byte a worker receives was sent by someone. Compare how many bits must be received in total with how much sending and receiving capacity the whole fleet has per second. ``` #### What This Part Should Cover - Unit conversion and the time for a single full copy over one interface - The per-machine receive limit and the source's send limit - The fleet-wide capacity argument, and why it changes under the shared-bandwidth reading - Why the bound barely depends on the number of workers when peers can forward data ### Part 2 — The design Start from the naive design and explain its completion time. Evolve it into a design that approaches the lower bound, covering how the weights are split up, how each worker decides where to fetch each piece from, how a coordinator (if any) works, and how you verify that each worker's copy is correct. ```hint Make every receiver a sender Idle upload capacity on the workers is the resource the naive design wastes; think about how a worker can start forwarding before it has the whole file. ``` #### What This Part Should Cover - The naive baseline and why it scales with the number of workers - Chunking, pipelining and peer-to-peer or tree distribution, and how close each gets to the bound - Coordination: chunk assignment, peer selection, and avoiding a central bottleneck - Integrity checking per chunk and for the whole model ### Part 3 — Operating it Describe how the system handles slow or failed workers during a transfer, workers that join late, a corrupted chunk, and the switch from the old model to the new one without downtime. ```hint Design for the slowest machine The rollout finishes when the last worker finishes; ask what stops one slow or dead machine from holding everyone else up. ``` #### What This Part Should Cover - Straggler handling and retry without restarting from zero - Late joiners and node failures in the middle of a transfer - Rollout and cutover to the new model, including rollback - The metrics that show how the distribution is progressing ### What a Strong Answer Covers - A correct, clearly stated lower bound under each bandwidth interpretation - A step-by-step evolution from the baseline to a design near the bound, with a completion-time estimate at each step - Explicit trade-offs among chains, trees and swarms - Practical failure handling and verification ### Follow-up Questions - The fleet spans racks behind oversubscribed switches. How does topology awareness change the plan? - Only a fraction of each model changes between versions. How would you exploit that? - How would you load the weights into GPU memory while they are still downloading? - How does the design change if the source is an object store with very high aggregate bandwidth?