## Design Safe Distribution and Activation of Model Weights The source names model-weight deployment as the system-design topic but does not give a scale or service contract. For this practice version, design a system that takes an approved, immutable model release, distributes its weight artifacts to a serving fleet, loads them safely, and activates or rolls back a version without exposing partially loaded state. Clarify artifact size, sharding, fleet and accelerator topology, startup expectations, availability requirements, and whether several model versions must serve concurrently before selecting components. ### Part 1 — Publish an Immutable Model Release Define the release manifest, artifact storage, validation, approval, and APIs that move a candidate version toward deployment. #### What This Part Should Cover - Stable model, release, artifact, shard, format, and runtime-compatibility identities. - Content digests and size metadata for every immutable artifact. - An explicit release state machine rather than a mutable `latest` object. - Authorization and audit boundaries for publishing and approving a release. ```hint Bind intent to bytes A deployment should name a manifest whose artifact digests cannot change after approval. ``` ### Part 2 — Distribute Weights to the Serving Fleet Design the control plane and data path that place all required artifacts on eligible workers while limiting bandwidth, storage pressure, and correlated retries. #### What This Part Should Cover - Desired-state assignments and worker-reported download or cache state. - Resumable transfer, digest verification, bounded concurrency, and retry backoff. - Complete handling of sharded weights and local disk or memory capacity. - Reuse of verified cached artifacts without trusting a filename alone. ```hint Treat the manifest as the completion unit A worker is not ready merely because one shard downloaded successfully. ``` ### Part 3 — Load, Validate, and Shift Traffic Explain how a worker loads a release, proves readiness, joins a staged rollout, and returns to the prior version after a failed health or quality gate. #### What This Part Should Cover - Separate downloaded, loading, ready, active, draining, and failed states. - Compatibility and bounded smoke checks before the worker receives normal traffic. - Canary or incremental rollout with an explicit stop and rollback decision. - Safe coexistence, traffic switching, and in-flight request handling for two versions. ```hint Readiness is stronger than file presence The serving runtime must successfully construct and check the model before routing marks it available. ``` ### Part 4 — Reconcile, Secure, and Operate Handle controller failover, stale workers, interrupted rollouts, artifact corruption, access control, garbage collection, and observability. #### What This Part Should Cover - Idempotent commands, assignment generations, and reconciliation from durable desired state. - Fencing or rejection of stale activation commands after a rollout changes. - Least-privilege artifact access and no weight data in logs or status payloads. - Metrics for distribution, loading, readiness, rollout health, rollback, and capacity. ```hint Make rollout state recoverable A replacement controller should determine what is active from durable assignments and worker evidence rather than from the previous process's memory. ``` ### What a Strong Answer Covers - Separates immutable release publication, artifact distribution, runtime loading, and traffic activation. - Verifies complete content and runtime compatibility before declaring readiness. - Uses staged rollout, durable reconciliation, and a prepared rollback path. - Addresses bandwidth, storage, stale control messages, security, and measurable failure handling. ### Follow-up Questions 1. How would you prevent a fleet-wide download surge after a new release is approved? 2. What happens when a worker has every shard on disk but cannot allocate enough memory to load them? 3. How would you roll back if the previous version has already been evicted from some workers? 4. Which evidence should block promotion even when basic health checks pass?