Designing Safety, Quality and Cost Guardrail Metrics for a Gen AI Feature Launch
Company: Google
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
You are the product manager preparing to launch a feature built on a generative AI model. Before it goes live, which guardrail metrics would you put in place, and why does each one earn its place?
```hint Look past the success metric
Picture the launch hitting its main goal while something else quietly gets worse: for the people using the feature, for the people affected by what it produces, and for the business. Each of those failure modes needs a measure of its own.
```
### Clarifying Questions
- What does the feature do, and where: a stand-alone chat surface, generation inside an existing product such as email or documents, or an agent that can take actions for the user?
- Who can use it at launch, and are minors or regulated areas such as health or finance in scope?
- What primary success metric is the launch optimizing, so the guardrails can be chosen to catch what it might hide?
- Are the guardrails meant to gate the launch decision, to trigger rollback once live, or both?
### What a Strong Answer Covers
- A clear separation between the success metric and the guardrails, with each guardrail tied to a specific way a generative feature can fail
- Coverage across harmful output, factual accuracy, privacy and security (such as data leakage and prompt injection), fairness across groups, experience and reliability, and business health such as cost and effects on the core product
- How each guardrail is measured before launch, when there are no users yet, and after launch
- Thresholds, actions and owners: which metrics block launch, which trigger rollback, and which are only monitored
- The tension between guardrails, especially harmful output versus over-refusal, and the statistical limits of testing for rare failures
### Follow-up Questions
- Your harmful-output rate is within its threshold overall but much higher in one non-English language. Do you launch?
- How would you set a threshold for the factual error rate when no previous version exists to compare against?
- Which of your guardrails can be monitored automatically in real time, and which need human review?
Overview: A product manager metrics question on which guardrail metrics to set up before releasing a feature built on a generative model, and why each one matters. It tests how a candidate anticipates failure modes such as harmful or false output, privacy leaks, latency and cost, and turns them into measurable launch and rollback thresholds.