# Deploy a Large Model to GPU Workers Design a system that distributes a 500 GB model artifact to a fleet of 100 to 1,000 GPU workers. External ingress and each worker link are 10 Gbps. Minimize time until the fleet can serve while handling worker failures, retries, integrity verification, and repeated deployments. Compare direct fan-out, pipelined forwarding, and tree distribution, then explain how the design evolves toward 10,000 workers. ### Constraints & Assumptions - The model can be divided into independently verifiable chunks. - Workers may begin forwarding or persisting a chunk before the entire model arrives. - A worker must not serve a mixed or incomplete model version. - Network topology and aggregate bisection bandwidth may matter more than nominal link rates. - Deployment control must distinguish slow workers from a corrupted artifact or overloaded source. ### Clarifying Questions to Ask - Is there one external source link or several regional/object-store endpoints? - Do workers share a rack-local cache or peer-to-peer network? - How much local disk and prior-version retention is available? - Must all workers switch together, or can cohorts become ready independently? - Are model versions mostly new bytes or amenable to delta transfer? ### What a Strong Answer Covers - Bandwidth lower bounds and topology-aware chunk distribution. - Immutable versioning, verification, readiness, and atomic activation. - Backpressure, retry, and straggler handling without restart storms. - A credible comparison of direct, pipeline, and tree approaches at larger scale. ### Follow-up Questions 1. How would you deploy a small patch to an existing model safely? 2. How would you prevent peer distribution from becoming a security boundary violation? 3. What metrics reveal whether storage, network, or verification is the bottleneck? 4. How would you roll back after some workers have activated the new version?