Diagnosing and Reining In Runaway LLM Serving Spend for a Newly Shipped Feature

You are interviewing for a product manager role at Google. Soon after your team launched a generative AI feature, the cost of running the model behind it jumped well beyond what anyone expected. As the product manager, what levers can you pull to bring that cost back under control, and in what order would you use them? ```hint Find what moved before cutting Write the bill as a product of a few factors and check which one changed since launch; the right lever depends on which factor grew. ``` ### Clarifying Questions - Do we know yet whether the spike comes from more usage than forecast, or from each request costing more than planned? - Is the feature free to every user, or part of a paid plan whose revenue could absorb some of the cost? - Does the team pay per token for a hosted model, or serve its own model on reserved hardware? - Is there a hard budget the feature must meet, or a target cost per user it must reach to stay viable? ### What a Strong Answer Covers - A breakdown of the cost into its drivers and a quick diagnosis of which one caused the spike, before any cuts - Levers across product design, prompts and context, model choice, caching and infrastructure, and pricing or access limits - Sequencing by speed, savings and risk to quality, with fast containment kept separate from structural fixes - Guardrails showing that cost cuts are not degrading answer quality, latency or user value - Monitoring and launch practices that catch the next spike early ### Follow-up Questions - Leadership wants the cost down sharply within weeks, without pulling the feature. Which levers go first, and what do you accept in return? - A small share of users turns out to drive most of the spend. How do you tell whether that is abuse, a valuable power-user segment, or a product bug? - A cheaper model cuts cost but a quality metric dips slightly. How do you decide whether to ship the switch?