On-Call Case Study: Debugging a Misbehaving Production Server

You are the on-call engineer for a backend service, and you have just been paged about one of its production servers. The interviewer plays the role of the system and of your teammates: you learn what the dashboards, logs and commands show only by asking for them, so the quality of your questions drives the exercise. The specific incident used in the interview was not described. For practice, assume the page says that one server in a fleet behind a load balancer is returning slow and failed responses, and that nobody yet knows why. Talk through your investigation out loud. ### Constraints and Clarifications - Treat it as a live incident: user impact matters as much as finding the root cause. - You may ask for any metric, log line or command output, but ask for specific things rather than "everything". - You may mitigate before you fully understand the cause, but say what each action costs. ### Clarifying Questions - Is it only this server, or are other servers in the same fleet or region showing the same symptoms? - When did it start, and was there a deploy, configuration change or traffic change around that time? - What exactly does the alert measure (error rate, latency, a health check), and what are the normal values? - What does the service depend on (databases, caches, other services), and are those healthy? ### Part 1 — Triage and impact What do you do in the first few minutes after the page? Explain what you check first, how you decide how many users are affected, who you tell, and whether to mitigate before you understand the cause. ```hint Scope first Before reading any log on the box, find out whether the problem is this host, this service, or something they depend on, and what changed recently. ``` #### What This Part Should Cover - Acknowledging the page and establishing scope: one host, the whole fleet, or a dependency - Sizing user impact, and declaring an incident or escalating when warranted - Checking recent changes as prime suspects - A fast, low-risk mitigation option and its cost ### Part 2 — Find the cause Narrow the problem down on the server and in the service. Which signals and commands would you ask for, in what order, and how would you turn the answers into hypotheses you can confirm or rule out? ```hint Work through the resources For a single misbehaving host, think about each resource it can run out of and each thing it waits on, and what observation would confirm or rule out each one. ``` #### What This Part Should Cover - A systematic pass over host resources: CPU, memory, disk space and I/O, network, file descriptors and process state - Application signals: error logs, thread or connection pools, garbage collection, dependency latency - Explicit hypotheses, each tested with a specific observation - Comparison of the sick host with a healthy peer ### Part 3 — Mitigate, fix and follow through Once you have a likely cause, how do you restore service, confirm the fix worked, and make sure the problem does not come back? ```hint Restore, then preserve Think about which recovery actions destroy the evidence you need, and what you would capture before taking them. ``` #### What This Part Should Cover - Choosing between draining the host, restarting, rolling back or fixing forward, with trade-offs - Capturing evidence before destructive actions - Verifying recovery with the same signals that fired the alert - Communication during the incident and a blameless follow-up with concrete action items ### What a Strong Answer Covers - Curiosity and structure: specific questions, each with a reason, rather than random commands - Mitigating user impact first, with the root cause a close second - Fluency with common host and service diagnostics - Hypothesis-driven narrowing that updates as new information arrives - Clear communication and a follow-up that improves alerts, runbooks or safeguards ### Follow-up Questions - The host looks healthy on every resource, but requests to it are still slow. Where do you look next? - Restarting the process fixes the problem, and it comes back two days later. How do you proceed? - The same symptoms appear on all hosts at once instead of one. How does your investigation change? - What would you add so that the next on-call engineer finds this problem faster?