Design a system that distributes large machine learning model weight files to a fleet of GPU inference workers. A new model version is published as one or more immutable weight shards, and the system must get the **correct version** onto **thousands of workers across multiple regions** with staged rollout, integrity verification, access control, fast rollback, and minimal serving downtime. Produce an end-to-end design. At minimum, address: functional and non-functional requirements; high-level architecture; storage and metadata model; APIs for publishing / discovering / downloading / activating versions; how workers fetch and cache weights efficiently at this scale; versioning, consistency, and rollout strategy; and failure handling, security, monitoring, and scalability tradeoffs. ```hint Where to start Notice that two very different kinds of data are moving here: the weight bytes themselves, and the much smaller decision about which workers should run which version. Ask yourself whether those two have the same size, change frequency, durability, and consistency needs — and what it buys you if you stop conflating them into one system. ``` ```hint The bandwidth bottleneck Before picking a transport, estimate the bytes one release actually moves: roughly `model_size × number_of_workers`. Plug in the sizes and worker counts from the problem, then compare that figure to what a single central origin can plausibly serve several times a day. Let the size of that gap — whatever you compute — drive the rest of your fetch design, rather than reaching for a default like "everyone downloads from blob storage." ``` ```hint Safe activation The thing you most want to avoid is replacing a working model with a broken or half-downloaded one. Think about how to make "get the new bytes onto the host" and "start serving them" separable steps, what has to be true at each boundary before you commit traffic, and what you keep around in case it goes wrong. ``` ```hint Consistency model Consider how a freshly-restarted worker decides what it should be running. If the answer is "look at storage and pick the newest thing it finds," walk through what happens during an in-progress publish or an overlapping rollback. That failure mode should suggest where the authoritative answer needs to live and how a worker tells a newer instruction from a stale one. ``` ### Constraints & Assumptions State your assumptions explicitly; reasonable defaults for this problem: - A single model's weights total **tens to hundreds of GB** and may be split into multiple shards (e.g. tensor-parallel or pipeline shards, on the order of 8–64 shards of a few GB each). - New versions can be published **several times per day**; artifacts are write-once and immutable after finalize. - **Thousands of GPU inference workers** (assume $10^3$–$10^4$) spread across multiple regions and several GPU/hardware generations; each worker must end up on the version assigned to its service / region / hardware cohort. - Required capabilities: **staged rollout** (canary → ramp → full), **rollback**, **integrity verification**, **access control**, and **minimal serving downtime** during an upgrade. - Workers may restart, lose network mid-download, or join the fleet at any time and must converge to the correct version. - Discovery/metadata lookup must stay available even during a regional storage incident; serving must not stop while a new version is rolling out. - At least the last few known-good versions are kept warm (locally and in cache) for fast rollback. - Assume an internal object store and the ability to run an agent process on every inference host. ### Clarifying Questions to Ask - What is the target time-to-full-fleet for a release, and the rollback SLO once a bad version is detected — hard SLA or best-effort? - Are shards content-addressed (so identical shards across versions dedup), or is each version fully distinct bytes? - Push or pull — can the control plane reach workers, or do workers poll? What network-reliability and egress-cost constraints exist per region? - How sensitive are the weights — do we need encryption at rest, signed artifacts, and per-service authorization, or just internal access control? - What does "activation" require of the inference server — a process restart, an in-process hot-swap, or a blue/green replica? Can a host hold two versions resident at once? - What rollout-health signals exist (latency, error rate, GPU OOM, load failures, model-quality metrics), and which should auto-pause a rollout? ### What a Strong Answer Covers - Clean split between **immutable artifact storage** and a **mutable rollout control plane**; immutability of finalized versions. - A **metadata/registry model**: model + version records, shard list with sizes and **checksums**, manifest checksum, lifecycle states, hardware constraints, lineage. - **Distribution at scale**: an explicit transfer-cost estimate that justifies regional caches / CDN / hierarchical or P2P fan-out, chunked + **resumable** downloads, content-addressed **dedup**, prewarming. - A **safe worker activation flow**: stage → verify → shadow-load → health-check → atomic swap → retain previous for rollback. - **Rollout strategy**: canary → ramp → full driven by metadata, automatic pause on error-budget breach, fast metadata-only rollback. - **Failure handling**: corrupt/partial downloads, mid-rollout crashes, stale assignments, cache misses, region isolation. - **Security**: authn for publishers and workers, short-lived signed URLs, encryption, signature/checksum verification, audit trail. - **Observability**: per-version fleet counts, download/activation success, cache hit rate, bytes-by-tier, rollout progress, and serving metrics straddling activation. - **Tradeoffs** stated explicitly (CDN vs P2P, push vs pull, eager vs lazy fetch). ### Follow-up Questions - A region's cache fleet goes cold (eviction or outage) right as a 150 GB release ramps there. How does origin survive the thundering herd, and how do you bound origin egress? - Two rollouts overlap: a canary of v5 is ramping when an emergency rollback to v3 is triggered. How does a worker that is mid-download of v5 resolve the conflict correctly? - The new version passes shard checksums but produces subtly degraded output quality, and infra health metrics look green. How would the rollout system catch this before full-fleet exposure? - How would you extend the design so two versions share most shards (e.g. a LoRA/adapter delta on a shared base), to cut both transfer and disk?