Monitoring and Alerting for a Service You Operate as the Sole On-Call Engineer

You would be the only engineer in a US office who deploys and develops a cloud service and keeps it running. The service runs in the company's own US data center, while the rest of its development team works in another country and time zone and is assumed not to have access to the US production environment. Every production problem therefore lands on you, and you are on call. The interviewer focused on monitoring: how you would set it up for this service, and which decisions monitoring forces on you in practice. The interview also probed unexpected problems you had hit while deploying cloud services. The service's architecture, traffic and existing tools were not described, so state your assumptions about them. ### Constraints and Clarifications - Not given: what the service does, its architecture, its traffic, any availability target, or the monitoring tools already in place. ### Clarifying Questions - What does the service do, who calls it, and what does a failure look like to them? - What does it run on: virtual machines, containers on Kubernetes, managed databases? Which monitoring and paging tools exist today? - Is there an availability or latency target agreed with the business? - Which logs and metrics may be shared with the overseas team, and can they help at all during an incident? - Is there any second line of support in the US, or a vendor for hardware and network problems in the data center? ### Part 1 — Set up monitoring for the service Describe what you would monitor for this service, how the data is collected and kept, and how alerts reach you. ```hint Start from the user, not the host Before listing host metrics, decide what it means for this service to be working for its callers, and how you would know within minutes that it is not. ``` #### What This Part Should Cover - Signals at several layers: the service as its users see it, its dependencies, the platform underneath, and the deployment process - Collection, retention and dashboards, including how the monitoring itself is monitored - Alert routing and escalation for a single on-call engineer ### Part 2 — The decisions monitoring forces in practice Walk through the decisions you actually face once the monitoring exists: what pages you at night and what waits until morning, how thresholds are chosen and revised, and how you keep alerts from turning into noise. ```hint One person's pager Every page interrupts the only person who can respond. For each alert, ask what action it requires and how soon, and what that implies for whether and how it pages. ``` #### Clarifying Questions for this Part - How many off-hours pages a week would be acceptable to you and to your manager? - Are automatic actions such as restarts and rollbacks allowed in production without a person approving them? #### What This Part Should Cover - Paging versus ticket-level alerts, tied to user impact and the time a response takes - How thresholds and time windows are set, revised and kept from flapping - Noise control, automation, and the cost of telemetry - Working across the time zone gap with developers who cannot see production ### Part 3 — An unexpected problem during a cloud deployment The interviewer asked about unexpected problems you had hit while deploying cloud services. Pick one, and walk through how it was detected, diagnosed and resolved, and what you changed in monitoring or in the deployment process afterwards. ```hint Detection is part of the story Say how the problem was first noticed: by an alert, by a user, or by you watching. That answer connects the incident to Part 1. ``` #### What This Part Should Cover - A concise situation and your specific role in it - Detection, mitigation and diagnosis, with how long each took - The lasting fix, and what changed in monitoring, alerting or the deployment process ### What a Strong Answer Covers - Monitoring designed from user impact downward, with a short list of actionable pages - Recognition that a one-person on-call makes alert quality, runbooks and automation essential - Monitoring that survives the failures it should report, through checks from outside the data center and a heartbeat alert - Deployment safety tied to monitoring: health checks, canaries and automatic rollback - A concrete, honest incident account with measurable impact and lasting changes ### Follow-up Questions - Your monitoring runs in the same data center as the service, and the data center loses its network. How do you find out? - Nobody has set an availability target for this service. How would you choose one? - A deploy passes every health check, but an hour later a caller reports wrong results. What monitoring would have caught it? - How would you reduce your on-call load over your first three months?