Walk Through Your Most Memorable Production Incident and the Commands You Ran

The first-round interview for a site reliability engineering role was a deep dive into the candidate's past projects. The central question: describe the most memorable production incident you handled and how you dealt with it. While you tell the story, the interviewer stops you to ask which commands you ran to investigate, what their output showed, and what you concluded from it. ### Clarifying Questions - Should the incident be one you led, or is one where you were among several responders acceptable if you explain your own part? - How much background on the system's architecture do you want before the incident itself? - Some of the tools were internal to a previous employer. Should I describe them by what they do? ### Part 1 — Tell the incident story Choose one incident and walk through it from the first signal to the follow-up work: the system and your role, the impact, how it was detected and triaged, how you mitigated it, how you found the root cause, and what changed afterwards. ```hint Lead with impact Before any detail, give the scale: who was affected, how badly, for how long, and how you found out. ``` #### What This Part Should Cover - The system, your role and the user impact, stated briefly with concrete measures - A clear timeline that separates mitigation from root-cause analysis - The decisions you made under pressure, and why - Follow-up actions and how you confirmed that they worked ### Part 2 — Show the investigation at command level Now the interviewer asks at each step: which command did you run, what did its output show, and how did that narrow down the cause? Walk through the investigation command by command, saying what each output looked like and what it ruled in or out. ```hint One hypothesis per command For each command, name the hypothesis it was meant to confirm or rule out, and the field of its output you looked at. ``` #### What This Part Should Cover - A logical order of investigation, from service-level symptoms down to host, process, network and kernel - Accurate knowledge of each command's output and of what normal and abnormal values look like - Correlating command output with logs, metrics and the timing of changes - What a command could not show, and what you used instead ### What a Strong Answer Covers - A real, specific incident told with structure and measurable impact - Command-level fluency that holds up under repeated "what did it show?" questions - A reasoned choice between rolling back, failing over and continuing to diagnose - Blameless follow-up: fixes, alerts and runbook changes that prevent a repeat - Honest attribution of what you did and what the team did ### Follow-up Questions - If the same failure started again tomorrow, which alert would catch it earlier, and on which signal? - The load average is high but CPU utilization is low. What can cause that, and which commands tell the causes apart? - How would you have investigated with metrics and logs only, without shell access to the hosts? - How did you decide when to stop investigating and roll back instead?