When to automate operations work, and good-enough versus excellent automation

In a senior reliability engineering loop, the interviewer asks how you think about automating the operation of a system so that engineers win back time for development work. They then push on two points: what separates automation that is merely good enough from automation that is excellent, and how you decide that something is not worth automating at all. There is no system to draw; the interviewer wants a principled position backed by concrete examples. ### Clarifying Questions - Should I focus on operational work for a live service (deployments, remediation, capacity changes), or include developer workflow tooling as well? - How large is the team that would build and then maintain the automation? - Is there existing data on where on-call and operational time goes? ### Part 1 — Your automation philosophy Explain how you choose what to automate first, how you build automation that is safe to run against production, and how you confirm that it actually gave engineers time back. ```hint Measure before you build Think about how you would find out which manual work costs the most, and how automation itself can become a new source of incidents. ``` #### What This Part Should Cover - How operational toil is identified and measured - Prioritization by frequency, effort and risk - Safety properties for automation that changes production - Evidence that time was freed, not just moved elsewhere ### Part 2 — Good enough, excellent, and where to stop Describe your bar for good-enough versus excellent automation, and where you draw the line on what is worth automating. ```hint Let the blast radius decide Consider how often a task runs and how much damage a mistake can do, and how those two things should change the level of polish you require. ``` #### What This Part Should Cover - Explicit criteria separating good-enough from excellent - A cost-benefit test for whether to automate at all - Cases where removing the work beats automating it - Concrete examples on both sides of the line ### What a Strong Answer Covers - A clear, defensible position rather than "automate everything" - Toil measured and prioritized with simple arithmetic - Automation treated as production software: tested, observable, owned - Polish proportional to frequency, blast radius and who runs it - Awareness of the risks automation introduces, including skill atrophy and automation that hides problems ### Follow-up Questions - An automation you built caused an incident. What do you do next? - How do you keep on-call engineers able to act manually when most remediation is automated? - How would you convince a product-focused team to spend a sprint removing toil?