How You Build Service Stability and Troubleshoot Production Incidents

In a hiring-manager round for a senior site reliability engineering (SRE) role, you are asked about your reliability practice: how you make the services you are responsible for stable, and how you troubleshoot an incident when one happens. Answer with concrete mechanisms from systems you have operated. ### Constraints and Clarifications - Assume the interviewer may ask no follow-up questions, so each answer should be complete and structured without prompting. - Anchor the answers in systems you have operated, and describe their type and scale without confidential details. - Treat "stability" as availability, latency, correctness, and safe change for production services. ### Clarifying Questions - What kind of systems does the team run: user-facing online services, infrastructure platforms, or data pipelines? - Does the role own on-call and incident command, or mainly reliability tooling and reviews? - Are reliability targets already defined for the team's services, or would the new hire help define them? ### Part 1 — Building Stability How do you do stability work for the services you are responsible for? ```hint Start from how failures reach users Consider the main ways production breaks, such as bad changes, capacity limits, failing dependencies, and slow detection, and name the mechanism you rely on for each. ``` #### What This Part Should Cover - Measurable reliability targets and how they guide priorities. - Change safety: how releases and config changes are rolled out and rolled back. - Resilience and capacity: redundancy, timeouts, overload protection, and planning. - Detection and learning: alerting on user-visible symptoms, drills, and postmortems. ### Part 2 — Incident Troubleshooting When an incident occurs, how do you troubleshoot it? ```hint Decide what comes first Consider what should happen first while users are still affected, and how you would narrow the scope before searching for a root cause. ``` #### What This Part Should Cover - Triage: severity, roles, and communication. - Stabilizing the service and choosing between rollback, failover, and other mitigations. - A systematic way to narrow down the cause using recent changes, scope, and telemetry. - Recovery checks, the postmortem, and follow-through on action items. ### What a Strong Answer Covers - Specific mechanisms and at least one real example, not a list of practices. - Priorities driven by user impact: restore service first, then find the root cause. - Explicit trade-offs, such as reliability against delivery speed, alert sensitivity against alert fatigue, and rollback against a fix deployed in place. - Senior-level ownership: setting standards across teams and making sure lessons are acted on. ### Follow-up Questions 1. How do you choose a service's reliability target, and what happens when its error budget runs out? 2. How would you investigate a latency increase that affects only some requests when nothing was deployed recently? 3. How do you make sure postmortem action items are completed rather than forgotten?