Respond to a Safety Regression before Releasing a Model

A model is approaching release, but evaluation indicates that its safety performance has declined. How would you investigate the regression, decide whether to release, and address the problem? ### Constraints and Clarifying Questions The model type, safety metric, affected users, severity, release date, and acceptable thresholds are unspecified. Ask what worsened, which version is the comparison baseline, and what evidence established the decline. State any scenario used to illustrate your answer as a practice assumption. Do not assume an actual company's release policy or a universal safety score. ### What a Strong Answer Covers A concrete investigation from evaluation validity to likely causes; treatment of severity, uncertainty, and affected subgroups; mitigation and regression testing; and an explicit release decision with monitoring and recovery criteria. ### Follow-up Questions - What if the aggregate safety score improves while one high-severity category worsens? - What if a serving-layer mitigation restores the measured score but substantially reduces useful responses? - How would you distinguish a model regression from an evaluator or prompt-distribution change?