Using AI Agents and Production Metrics to Find and Verify Efficiency Improvements

The interviewer introduces a hypothetical scenario: AI agents analyze an existing production system and recommend improvements to its efficiency. Discuss how you would make this work and how you would judge the results. The scenario does not name the system or define efficiency, so settle those points first; you may base your answer on a system you know well, such as one you built. ### Clarifying Questions - What kind of system is it: a request-serving service, a data pipeline, batch jobs, or a mix? - Does efficiency mean cost, resource utilization, latency, throughput, or a combination, and which of these must not get worse? - What telemetry already exists, and what may the agents read? - Are the agents only advisers, or may they apply changes themselves, and if so, which kinds? ### Part 1 — Metrics and instrumentation What metrics would you collect to evaluate the system's efficiency, and how would you instrument the system to expose those metrics? ```hint Efficiency per unit of work Raw CPU or latency figures say little until you relate them to the work done; think about which denominators make two time periods comparable. ``` #### What This Part Should Cover - Work, health, resource and cost metrics, with efficiency expressed per unit of work - Instrumentation with shared attributes, such as service, route and deploy, so that signals can be joined - Recording change events, and keeping the cost and cardinality of telemetry under control ### Part 2 — From metrics to recommendations How could AI agents use these metrics to identify inefficiencies, and what kinds of improvements might they recommend? Give a concrete example of an optimization an AI agent could suggest, with the evidence behind it. ```hint Evidence, not dashboards Consider what an agent needs beyond the numbers, such as traces, query plans, code and recent deploys, to turn an anomaly into a specific recommendation someone can check. ``` #### What This Part Should Cover - A detection, investigation and recommendation loop, with the tools and access the agents need - The range of improvements: capacity, data access, caching, batching, storage and scheduling - A concrete example backed by specific evidence - Controls against wrong or unsupported recommendations ### Part 3 — Did it actually help? How would you evaluate whether a recommendation, once applied, actually improved the system? ```hint Before and after is not enough Ask what else changes between last week and this week besides the recommendation, and how you would separate its effect from those changes. ``` #### What This Part Should Cover - A success metric and guardrails fixed before the change - Normalization and a concurrent control instead of a plain before-and-after comparison - Duration, attribution and rollback ### What a Strong Answer Covers - Efficiency defined as resources or cost per unit of useful work, with latency and error-rate guardrails - Instrumentation that lets metrics, traces, profiles and costs be joined by service, route and deploy - A grounded agent loop: evidence gathering, specific recommendations and expected impact that can be checked - Human approval and safe rollout, with agents read-only on production by default - Rigorous evaluation with controls, guardrails and rollback, and a measure of whether the agents themselves are worth running ### Follow-up Questions - An agent recommends halving a service's memory limit based on two weeks of usage data. What could go wrong, and what would you check first? - How would you stop agents from proposing the same rejected recommendation every week? - Two recommendations ship in the same week and costs fall. How do you attribute the improvement? - What, if anything, would you let an agent change without human approval?