ChatGPT Metric Diagnosis: Why Are More Responses Getting Negative Ratings?

You are a product manager working on ChatGPT, and you notice that the share of responses receiving a thumbs-down rating has gone up. How would you work out the root cause, and what would you do about it once you know? ```hint Trust the number before explaining it A thumbs-down rate is a ratio, and both its numerator and its denominator can move for reasons unrelated to answer quality. Confirm the change is real and find where it is concentrated before forming hypotheses about the model. ``` ### Clarifying Questions - How is the metric defined: thumbs-down ratings per response shown, or per response that received any rating? Has the share of responses that get rated changed as well? - How large is the increase, and did it appear suddenly on one day or build up gradually? - Is it concentrated anywhere: a particular model, platform, language, country, plan (free or paid), or feature such as file uploads, web search or voice? - What changed around that time: a model update, a change to system instructions or safety policy, a change to the rating buttons, or a competitor launch? ### What a Strong Answer Covers - Validating the metric first: its definition, logging and client changes, changes to the rating UI, and shifts in who rates - A structured hypothesis tree (measurement, internal change, user mix, external events) checked in a sensible order, with the segment cuts that separate the branches - Qualitative evidence, such as feedback reasons, free text and sampled conversations, to learn what users dislike - A fix matched to the confirmed cause (rollback, targeted model or policy change, product change), validated before full rollout - Companion metrics and guardrails, since thumbs ratings are a noisy and biased proxy for answer quality ### Follow-up Questions - Segmentation shows the rise is limited to long conversations that include file uploads. What would you investigate next? - The rise started right after a model update that improved most offline evaluations. Do you roll it back? - Why could optimizing the model directly for fewer thumbs-down ratings make the product worse, and how would you guard against it?