Design a System to Distribute Large ML Model Files Across a Serving Fleet

Design a system that distributes large model files, such as the trained weights of machine-learning models, to the machines that serve them. A training or export pipeline publishes a new version; a large fleet of serving hosts, possibly spread across several data centers, must receive it, verify it and switch to it, and must be able to switch back if the new version misbehaves. The prompt was given only as a title, so treat sizes and fleet shape as open questions and state your assumptions. The core difficulty is that many hosts want the same very large files at the same moment. ```hint Count the bytes Multiply the file size by the number of hosts and compare the result with what a single origin can serve in your time budget; that comparison decides the shape of the design. ``` ```hint Look for copies that already exist Think about where else a host could fetch the data from once some machines near it already hold part of it. ``` ### Constraints and Clarifications - A model version is immutable once published; a fix is a new version. - Assume serving hosts have local disks with room for a small number of versions, and confirm how many. - Distribution traffic must not degrade the live inference traffic that shares the same hosts and network. - Training and the internals of the serving process are out of scope; the design ends when the serving process is told that a verified version is on disk and ready to load. ### Clarifying Questions - How large is one model version, how many hosts need it, and across how many data centers or regions? - How quickly must a new version reach the whole fleet, and must it go to a canary group first? - How often are versions published, and does each host serve one model or several? - What are the bandwidth limits: host network interfaces, links between data centers, and read throughput from the central store? - How fast must a rollback be? ### What a Strong Answer Covers - Back-of-envelope reasoning about total bytes, origin capacity and time to reach the fleet that justifies the distribution topology - A fan-out strategy that avoids overloading the origin and the links between data centers - Versioning, immutability and integrity verification before a model is loaded - Rollout control: staged rollout, an atomic switch on each host, fast rollback, and management of old versions on disk - Failure handling for slow or failed hosts, interrupted downloads, and outages of shared components - Visibility into distribution progress and per-host state ### Follow-up Questions - How would you shorten the time from "version published" to "serving traffic" when loading the weights into accelerator memory is also slow? - Only a small part of the weights changes between consecutive versions. How could the system take advantage of that? - How do you make sure a corrupt or tampered artifact can never reach production? - The link between one region and the origin fails in the middle of a rollout. What happens?