Design Model Weight Distribution

Design a system for distributing large machine learning model weight files to a fleet of inference workers. Context: - Model weights may be tens to hundreds of GB and may be split into multiple shards. - A new model version can be published several times per day. - Thousands of GPU inference workers across multiple regions need to receive the correct version. - The system must support staged rollout, rollback, integrity verification, access control, and minimal serving downtime. Discuss: - Functional requirements and non-functional requirements. - High-level architecture. - Storage and metadata design. - APIs for publishing, discovering, downloading, and activating model versions. - How workers fetch and cache weights efficiently. - Versioning, consistency, and rollout strategy. - Failure handling, security, monitoring, and scalability tradeoffs.