Stop Treating AI Models Like People

No, they haven’t decided to teach themselves anything, they don’t love you back, and they still aren’t even a little bit sentient.

By Sasha Luccioni and Gary Marcus

The ELIZA effect is still with us, more than half a century later

For the last few months, people have had endless “conversations” with  chatbots like GPT-4 and Bard, asking these systems whether climate change is real, how to get people to fall in love with them, and even their plans for AI-powered world domination. This is apparently done by operating under the assumption that these system have genuine beliefs, and the capacity to teach themselves, as in this Tweet from the US Senator Chris Murphy:

In the language of cognitive psychology, all of this is “overattribution”, ascribing a kind of mental life to these machines that simply isn’t there, like when many years ago people thought that Furbies were learning language, when in reality the unfolding of abilities  was pre-programmed. As most experts realize, the reality is that current AI doesn’t “decide to teach itself”, or even have consistent beliefs. One minute the string of words that it generates may tell you that it understands language.

And another it may say the opposite.

There is no there there, no homunculus inside the box, no inner agent with thoughts about the world, not even long-term memory. The AI systems that power these chatbots are simply systems (technically known as “language models” because they emulate (model) the statistical structure of language) that compute probabilities of word sequences, without any deep or human-like comprehension of what they say. Yet the urge to personify these systems is, for many people, irresistible, an extension of the same impulse that makes see a face on the Moon or attributing agency and emotions to two triangles “chasing” each other around a screen. Everyone in the AI community is aware of this, and yet even experts are occasionally tempted to anthropomorphism, as deep learning pioneer Geoffrey Hinton’ recently tweeted that “Reinforcement Learning by Human Feedback is just parenting for a supernaturally precocious child.”  Doing so can be cute, but also fundamentally misleading, and even dangerous.

The fact that people might over attribute intelligence to AI system has been known for a long time, at least back to ELIZA, a computer program from the 1960s that was able to have faux-psychiatric conversations with humans by using a pattern matching approach, giving users the impression that the program truly understood them. What we are seeing now is simply an extension of the same “ELIZA effect”, 60 years later, where humans are continuing to project human qualities like emotions and understanding onto machines that lack them. With technology more and more able to emulate human responses based on larger and larger samples of text (and “reinforcement learning” from humans who instruct the machines), the problem has grown even more pernicious. In one instance, someone interacted with a bot as if it were somewhere between a lover and therapist and ultimately committed suicide; causality is hard to establish, but the widow saw that interaction as having played an important role; the risk of overattribution in a vulnerable patient is serious.  

As tempting as it is, we have to stop treating AI models like people. When we do so, we amplify the hype around AI, and lead people into thinking that these machines are trustworthy oracles capable of manipulation or decision-making, which they are not. As anyone who has used these systems to generate a biography is aware of, they are prone to simply making things up; treating them as intelligent agents means that people can develop unsound emotional relationships, treat unsound medical advice as more worthy than it is, and so forth. It’s also silly to ask these sorts of models for questions about themselves; as the mutually contradictory examples above make clear, they don’t actually “know”; they are just generating different word strings on different occasions, with no guarantee of anything.) The more false agency people ascribe to them, the more they can be exploited, suckered in by harmful applications like catfishing and fraud, as well as more subtly harmful applications like chatbot-assisted therapy or flawed financial advice. What we need is for the public to learn that human-sounding speech isn’t actually necessarily human anymore; caveat emptor. We also need new technical tools, like watermarks and generated content detectors, to help distinguish human- and machine-generated content, and policy measures to limit how and where AI models can be used.

Educating people to overcome the overattribution bias will be a vital step; we can’t have senators and members of the AI community making the problem worse. It is crucial to retain a healthy skepticism towards these technologies, since they are very new, constantly evolving, and under-tested. Yes, they can generate cool haikus and well-written prose, but they also constantly spew misinformation (even about themselves), and cannot be trusted when it comes to answering questions about real-world events and phenomena, let alone to provide sound advice about mental health or marriage counseling.  

Treat them as fun toys, if you like, but don’t treat them as friends.

Dr. Sasha Luccioni is a Researcher and Climate Lead at Hugging Face, where she studies the ethical and societal impacts of AI models and datasets. She is also a Director of Women in Machine Learning (WiML), founding member of Climate Change AI (CCAI), and Chair of the NeurIPS Code of Ethics committee.

Gary Marcus (@garymarcus), scientist, bestselling author, and entrepreneur, is deeply, deeply concerned about current AI but really hoping that we might do better.

Watch for his new podcast, Humans versus Machines, debuting April 25th, wherever you get your podcasts.