Design a Deployment and Orchestration System for a Large Server Fleet

Design a system that deploys software to a large fleet of servers and orchestrates the rollout. The report gives only this one-line prompt, so settle the scope with the interviewer first. In its usual form, an engineer submits a new version of a service, and the system rolls it out to the hosts that run the service in controlled stages, checks health as it goes, and stops and rolls back when the new version misbehaves. ```hint Commands or desired state Decide whether the control plane sends commands to hosts or records the version each host should run and lets each host converge to it, and what each choice means for a host that is offline during a rollout. ``` ```hint Limit the blast radius Decide how a rollout is split into stages, how many of a service's hosts may be out of service at once, and what evidence moves a rollout from one stage to the next or stops it. ``` ### Constraints and Clarifications - Not given: fleet size, what is deployed (binaries, containers, configuration), deployment frequency, whether the fleet spans several regions, and how quickly a rollout must finish. Ask, or state your assumptions. ### Clarifying Questions - What is deployed: packages or binaries on long-lived hosts, container images, configuration, or all of these? - How many hosts and services are there, how many deployments a day, and across how many regions and zones? - Does orchestration mean ordering and pacing the rollout, or also deciding which hosts run which service and keeping a target number of healthy instances? - What health signals exist (host health checks, error rates, latency alarms), and should rollback be automatic? - How much of a service's capacity may be out of service during a deployment? - Can two deployments of the same service overlap, and what should happen to a host that cannot be reached? ### What a Strong Answer Covers - Scopes the problem and sizes the fleet and deployment rate before designing - A split between a control plane and a host agent, with the desired version stored durably and hosts reporting their actual state - Staged rollout with bounded batches, health gates between stages, and automatic halt and rollback - Correct behavior when hosts, agents or the orchestrator fail mid-rollout, and when deployments of one service collide - Distributing artifacts to many hosts at once, and scaling the control plane - Auditability, observability, and safety controls such as pausing, approvals and freezes ### Follow-up Questions - Halfway through a rollout, error rates rise in one zone only. What does your system do, and how does it tell a bad release from a bad zone? - Thousands of hosts download a large artifact at the same moment. How do you avoid overwhelming the artifact store? - The orchestrator crashes in the middle of a rollout. How does it resume without redeploying or skipping hosts? - How do you safely deploy a new version of the deployment agent that runs on every host?