Live troubleshooting: diagnose and mitigate very high CPU on a public-facing service

This is an interactive scenario from a senior reliability engineering loop. You are told that a public-facing service is running at very high CPU usage. The interviewer then steps through the situation with repeated "what would you look at next?" prompts and reveals information only when you ask for it. Walk through how you would confirm and scope the problem, narrow down the cause, mitigate it, and follow up. One mitigation that comes up is rate limiting customers. Be ready to defend or reject it for a service where every API call carries revenue and customer trust, compared with scaling out or moving work to an asynchronous queue. ### Constraints and Clarifications - You do not get the full picture up front; part of the evaluation is which questions you ask and in what order. - Assume the usual tools are available: dashboards for service and host metrics, logs, the deployment and configuration history, and the ability to profile a running instance. ### Clarifying Questions - Is CPU high on every instance, or only on some (one region, one build version, one host type)? - Are customers affected yet: are latency and error rates up, or is it only the CPU metric? - Did anything change recently: a deployment, a feature flag, configuration, or a dependency? - Has request volume changed, and is any increase concentrated in a few customers or endpoints? - Does the service autoscale, and is it stateless? ### Part 1 — Confirm and scope What do you check in the first few minutes, and how do you decide how urgent this is? ```hint Symptom versus cause High CPU describes a possible cause; decide which customer-facing symptom tells you how urgent it really is. ``` #### What This Part Should Cover - Validation of the signal and of its customer impact - Scoping by instance, region, version and time of onset - A severity call and who needs to know ### Part 2 — Narrow the cause Walk through what you would look at next, and how each answer changes your next step. ```hint More work or costlier work? Split the possibilities into more requests arriving versus each request costing more, and decide what evidence separates the two. ``` #### What This Part Should Cover - A branching investigation rather than a fixed checklist - Use of change history, traffic breakdowns and profiling - Common causes of CPU saturation and how each one is recognized ### Part 3 — Mitigate Choose a mitigation. Compare rate limiting customers, scaling out, and moving work to an asynchronous queue, for a service where rejected calls cost revenue and trust. ```hint Who pays for each option? For each mitigation, ask which customers are affected, how fast it takes effect, and what it does to downstream dependencies. ``` #### What This Part Should Cover - Rollback when a change is the cause - Trade-offs among scaling out, targeted throttling, broad rate limits and asynchronous processing - Protecting downstream dependencies and avoiding a retry storm ### Part 4 — After the incident What do you do once the service is stable? ```hint Make it boring next time Think about what would have caught this earlier, or absorbed it automatically. ``` #### What This Part Should Cover - Root cause confirmation and a blameless review - Capacity, autoscaling and quota changes - Detection and alerting improvements ### What a Strong Answer Covers - Targeted questions in a sensible order that adapt to each answer - Customer impact first, then cause - A clear distinction between load increases and per-request efficiency regressions - A mitigation chosen with explicit trade-offs, including the business cost of rejecting traffic - Follow-through that prevents recurrence ### Follow-up Questions - Scaling out does not lower CPU per instance. What does that tell you, and what do you do next? - The spike comes from one large customer's legitimate batch job. How do you handle it without damaging the relationship? - Profiling shows most CPU time in garbage collection. What do you investigate? - How would you design alerting so this is caught before customers notice?