Design Batch Reboots for Machines
Company: Coreweave
Role: Site Reliability Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
Design a production system that can safely batch reboot `N` machines in a fleet.
Context: You operate a large fleet of machines used for production workloads. Operators need a reliable way to reboot many machines, for example after kernel upgrades, hardware remediation, firmware updates, or node recovery. The system must avoid taking down too much capacity at once and must provide visibility into progress and failures.
Address the following:
1. How users submit a batch reboot request.
2. How the system selects and validates the target machines.
3. How to schedule reboots in safe batches or waves.
4. How to prevent service-impacting outages.
5. How to track machine state before, during, and after reboot.
6. How to handle failures, retries, timeouts, and partial completion.
7. How the system should integrate with infrastructure such as Kubernetes or a machine inventory service.
8. What observability, auditability, and safety controls are required.
Quick Answer: This question evaluates a candidate's competency in designing reliable, capacity-aware operational systems for large-scale machine management, including orchestration, failure handling, state tracking, observability, and integration with infrastructure services.