Risks of Anthropomorphizing Large Language Models in AI Products

Explain the risks of treating a large language model as if its thinking and feelings work the way a person's do. ```hint Trace one projection Take one trait people read into human speech, such as confidence or the reasons someone gives, and ask what goes wrong when you read a model's output the same way. Then ask who is exposed: users, product teams or evaluators. ``` ### Clarifying Questions - Should I focus on end users chatting with an assistant, or also on the teams that build, evaluate and deploy models? - Do you want thinking (knowledge, reasoning, memory) and feeling (emotions, experience) covered together, or mainly one of them? - Should I also address the opposite mistake, dismissing models as simple autocomplete, or stay with the risks of over-attribution? ### What a Strong Answer Covers - An accurate account of how language models produce text and why human-like language does not establish human-like cognition or experience, with honest uncertainty about open questions. - Concrete risks across groups: over-trust, emotional reliance, misread explanations, misjudged capabilities and blurred accountability. - Specific examples where human intuitions fail for models, such as fluent errors, uneven skills and unreliable self-reports. - Product and evaluation practices that keep people's mental models calibrated, weighed against the appeal of a natural, friendly interface. - Awareness of the cost of the opposite mistake. ### Follow-up Questions - Your assistant says things like "I'm so glad I could help." Would you remove that? How would you decide? - How would you measure whether users over-trust your product's answers? - A user tells support that the assistant is their closest friend and that they talk to it for hours every day. What should the product do?