Production Incident Response and Linux Troubleshooting Commands for an SRE

In a screen for a Site Reliability Engineer role, the interviewer asked how you handle a production incident and how you troubleshoot a failure, down to the specific commands you would use. Answer both for a service that runs on a fleet of Linux hosts. The report names these topics but no scenario. For practice, use this one: an alert fires because a backend service's error rate and response time have risen sharply, and one of its Linux hosts looks like the source. ### Clarifying Questions - Does the service run as several replicas behind a load balancer, so that one host can be taken out of rotation? - What monitoring is available: metrics dashboards, centralized logs, tracing? Do I have shell and root access on the hosts? - Did anything change recently, such as a deploy, a configuration push or a traffic shift? - Does the team have defined severity levels, on-call roles and an escalation path? ### Part 1 — Handle the incident Walk through what you do from the moment the alert fires until the incident is closed: what you check first, who you bring in, how you decide between mitigating now and investigating further, how you keep people informed, and what happens after service is restored. ```hint Restore or explain Ask what you would do if a deploy went out shortly before the alert: do you need the root cause before you act, and what would you want to save before you act? ``` #### What This Part Should Cover - Triage: confirming the alert, measuring user impact and scope, assigning a severity - Roles and communication: who coordinates, who investigates, how often updates go out, when to escalate - Mitigation before root cause, and the evidence to preserve first - Closing the incident: verifying recovery and a blameless postmortem with owned action items ### Part 2 — Troubleshoot the host You have shell access to the suspect host. Explain how you find what is wrong, naming the commands you would run at each step and what in their output you would look for. Cover CPU, memory, disk, network, the service process itself and its logs. ```hint Overview before detail Decide which one or two commands give you a picture of the whole host in the first minute, and what in their output tells you which resource to examine next. ``` #### What This Part Should Cover - A systematic order across resources rather than a list of unrelated commands - For each resource, a command and the specific output fields that show utilization, saturation or errors - Inspecting the process itself: its state, open files and sockets, system calls and logs - Turning a finding into a confirmed cause and a fix ### What a Strong Answer Covers - Restores service quickly while preserving the evidence needed to find the cause - Names concrete commands and explains how to read their output, not just their names - Separates symptoms from causes and confirms a hypothesis before acting on it - Keeps the cost of diagnostic tools on a struggling production host in mind - Communicates throughout and closes with a postmortem that prevents a recurrence ### Follow-up Questions - The load average is very high, but `top` shows the CPUs mostly idle. What is happening, and how do you confirm it? - `df` reports a filesystem as full, but `du` over the same filesystem adds up to far less. Why, and how do you free the space without restarting the service? - The service logs "too many open files", or outbound connections fail while thousands of sockets sit in TIME_WAIT. How do you investigate each? - The service process was killed and restarted with no error in its own log. Where do you look?