Patch Storage Data-Plane Hosts, Roll Out Fixes, Keep New Hosts on the Latest Image

You work on a storage service whose data plane runs on a fleet of hosts that store customer data and serve customer reads and writes. The hiring manager asked a series of short operational design questions about running such a fleet day to day; three of them were reported. Answer each one. ### Constraints and Clarifications - Not given: the kind of storage (block, object, file or database), how data is replicated across hosts, the fleet size, the patch cadence and the availability target. Ask, or state your assumptions. ### Clarifying Questions - Is customer data replicated across hosts, and across racks or availability zones, so that one host can go offline without losing availability or durability? - What kind of change is being rolled out: an operating system or kernel patch that needs a reboot, a firmware update, or a new build of the storage software? - Can a host be replaced instead of patched in place, and how long does it take to re-replicate one host's data elsewhere? - What counts as customer impact: failed requests, higher latency, or temporarily reduced redundancy? ### Part 1 — Patching hosts without customer impact How would you patch the data-plane hosts without impacting customers? ```hint What the host is holding Before a host goes down, think about everything it is currently responsible for, and where each of those responsibilities goes while it is away. ``` #### Clarifying Questions for this Part - May more than one replica of the same data be offline at the same time? #### What This Part Should Cover - Draining a host before work: traffic, leadership roles and new data placement - Limiting the blast radius by replica group and fault domain, with redundancy checks gating each step - Verification before a host returns to service, and the choice between patching in place and replacing the host ### Part 2 — Reliably deploying bug fixes How would you make sure a bug fix is deployed to the fleet reliably? ```hint Small first Think about the smallest exposure that would reveal a bad fix, and the automatic signal that lets you stop and roll back before more of the fleet is affected. ``` #### What This Part Should Cover - A staged rollout with canaries, waves and bake times - Automated health gates and a tested rollback - Compatibility while old and new versions run side by side, and the path for urgent fixes ### Part 3 — New hosts always run the latest image How would you make sure every newly spun-up host runs the latest image? ```hint Who decides what latest means Consider where the definition of the current image lives, and what stops a host from joining the fleet when it does not match. ``` #### What This Part Should Cover - A single source of truth for the approved image, resolved at launch time - An admission check before a new host stores data or serves traffic - Drift detection for hosts already running, and how this interacts with a rollout in progress ### What a Strong Answer Covers - Clarifies replication, fault domains and the meaning of customer impact before proposing procedures - Treats redundancy as the budget that maintenance spends, never letting any data drop below its safe replica count - Automated, gated, staged rollouts with fast rollback and compatibility across mixed versions - One source of truth for the desired image, enforced when a host joins and checked continuously for drift - Monitoring and safety stops that halt automation when something looks wrong ### Follow-up Questions - A critical security patch must reach the fleet much faster than the normal rollout allows. What changes in your process, and which safety checks do you keep? - A rollout is halfway done when you discover that the new version writes data in a format the old version cannot read. How do you roll back? - While a host is down for patching, another host holding replicas of the same data fails. What should the automation do? - How would you find out that patching is hurting customers in a way your health checks do not catch?