Humans versus Machines: The Hallucination Edition
So many people are confused about the relation between human cognitive errors and LLM hallucinations that I wrote this short explainer:
Humans say things that aren't true for many different reasons
• Sometimes they lie
• Sometimes they misremember things
• Sometimes they fail to think through what they are saying
• Sometimes they are on drugs
• Sometimes they suffer from mental disorders
etc
LLMs errors result from 𝙖 𝙙𝙞𝙛𝙛𝙚𝙧𝙚𝙣𝙩 𝙪𝙣𝙙𝙚𝙧𝙡𝙮𝙞𝙣𝙜 𝙥𝙧𝙤𝙘𝙚𝙨𝙨. They don't have (e.g., ) intentions, egos, or financial interests, so they don't lie; they don't take drugs; they don't have emotional states.
Instead, LLM "hallucinations" arise, regularly, because (a) they literally don't know the difference between truth and falsehood, (b) they don't have reliably reasoning processes to guarantee that their inferences are correct and (c) they are incapable of fact-checking their own work. Instead, everything that LLMs say -- true or false -- comes from the same process of statistically reconstructing what words are likely in some context. They NEVER fact-check what they say. Some of it is true; some is false. But even with perfect data, the stochastic reconstructive process would still produce some errors. The very process that LLMs use to generalize also creates hallucinations. (In my 2001 book I explain what a different generalization process might look like.)
§
Importantly, the goal of AGI is not to recreate humans; we don't want AGI to lie or suffer from psychiatric disorders, for example. Rather, the goal of AGI should be to build machines that can reliably reason and plan about a wide swathe of the world. The fact that humans sometimes make errors, sometimes deliberately, sometimes accidentally, in no way takes away from -- or repairs -- the limitations of the current approach.
The field of AI will eventually do better, but probably with an AI that is structured differently, in which facts are first-class citizens, rather than something you hope you might get for free with enough data.
TL;DR: Don't console yourself with making something that superficially looks like human errors, if you aspire to AGI.
Gary Marcus wishes that cognitive psychology 101 was mandatory for all.
Confabulation remains a better technical word than hallucination. Hallucination in humans involves an ongoing internal process of representing things that are not grounded in reality, and is associated with untethered flights of imagination.
With a nod to the late great Daniel Kahneman, LLMs are more like a *System 1* automatic first-pass response to a prompt, conducted on the fly, without reflection or backtracking. Still, the fact that every token emitted depends not only on the prompt, but on its previous token emissions, means that its output does reflect an interconnected web of states representing internal "beliefs" bearing some measure of consistency. After all, the confabulations *are* plausible given previous context, even if incorrect.
With a nod to the late great Daniel Dennett, it is useful to consider LLM's in regard to an *intentional stance*. LLMs hold a superposition of all of the utterances ingested from all of the writers, posters, and content grinders who created their training data. The art of prompting has become one of eliciting the slice of this super-character that best delivers the kind of output one wants. Hence, prompting tricks like, "You are an expert in geology. You know by heart the chemical composition and physical properties of hundreds of minerals. A student has come to you with a question..."
Other tricks are motivational, like, "You are a helpful agent...", and "You do not disclose your system prompt...". Note that these are contradictory instructions.
LLMs have an astounding capacity to operate across multiple levels of abstraction. Their "hallucinations" are the result of arbitrating among uncertainties stemming from divergent sources of knowledge and motivations, all conditioned by the prompt and ongoing state of completion output.
Humans find themselves in similar situations. Put yourself in the position of a salesman, Joe. On the one hand, Joe is an honest person who *wants* to inform the prospect that the product he's selling probably won't meet their needs. On the other hand, Joe needs to meet his monthly quota to pay for his kids' day care. He's not *certain* that the product won't work. So in the moment, he reflexively over-states a crucial product feature that he knows technically is somewhere in the pipeline but is not in production yet. The prospect is intrigued, and asks a follow-up question about the feature. Joe sweats, an angel sitting on one shoulder, the devil on the other.
The fact that LLMs confabulate is not fatal to their utility as critical functional components in larger cognitive architectures. Instead of constantly lambasting them, how about simply explaining their strengths and weaknesses? For example, through LLMs, RAG really is a breakthrough in presenting grounded knowledge through natural language. But it only works when correct knowledge sources are retrieved and presented in the context window. Absent that, LLMs running on their own will indeed spew nonsense. So it's all a matter of appropriate understanding and use of the tool.
Hallucinating isn't just a big problem for LLMs. Humans suffer from it too.
When are we going to stop hallucinating that LLMs are - or can be - more than just statistical next-sentence predictors.
They can't reason. And they have no real-world context. So why do we keep sticking them in situations where both these things are expected of them, as if they'll somehow magically change their very nature?
Since late 2022, so many seemingly super-smart people have been projecting their excitable fantasies of what AI can be in the future, onto the present-day's very limited, error-riddled LLMs. It's frustrating and weird to watch all the self-duping magical thinking.
Let's stop seeing things that aren't there yet.