“The definition of insanity is doing the same thing over and over again while expecting different results.” If all we had was ChatGPT, we could say, hmm “maybe hallucinations are just a bug”, and fantasize that they weren’t hard to fix. If all we had was Gemini, we could say, hmm “maybe hallucinations are just a bug”.
Another person commented on this substack a year ago, LLMs are doing the same thing when they get it right that they are doing when they get it wrong. They said it better, but I’ve never forgotten.
I had the original in my notes - was this yours? “When an LLM ‘hallucinates’, it’s doing the exact same thing it does when it tells the truth.” Would have been from September 2023 or shortly before.
"Which means that showing all these errors is fine, but labelling them ‘errors of understanding’ may have the opposite effect of what we realists want to achieve, as the underlying message is interpreted as: “the (real) understanding is there, it just goes wrong sometimes“. In reality there is no understanding to begin with, so that hidden, unintended, message is wrong.
The system calculates, and the ‘error’ is not an ‘error’ from the perspective of that calculation. What we see as an error is what the system has calculated as ‘the best’. It is ‘correct’. Exactly the same way as it has calculated the results we humans consider correct. We should therefore not say ‘the system (still) makes error of understanding’ but we should say ‘the system lacks understanding, so incorrect results are to be expected’. Do not talk about ‘errors’."
Another one that is one thing producing both the wanted and unwanted effect is 'memorisation' which can be something we like (GPT correctly quotes Shakespeare) and dislike (GPT correctly quotes copyrighted material — i.e. effectively data leakage)
Another one: "It is also good to repeat another perspective here. GenAI doesn’t produce ‘results’ and ‘hallucinations’. It produces ‘approximations’, period. Even its correct answers are approximations of understanding, not understanding itself." from https://ea.rna.nl/2024/02/13/will-sam-altmans-7-trillion-ai-plan-rescue-ai/ That one also contains some rough calculations about scaling (and how that is not going to solve the problem). Gary has also mentioned that both correct and incorrect results are both (good and bad) approximations: https://garymarcus.substack.com/p/sora-cant-handle-the-truth
Personally appreciate the approximations framing as well because it gets how genAI may be an additional layer distancing us from the pursuit of truth goal, rather than the promised ability to get to truth faster.
Science is also an approximation of understanding, not understanding itself. I don't think that framing is helpful. What y'all are trying to get at is some lack of deeper model, not the degree of truthfulness per se.
AI Numerologists have to say their word salad spewing stochastic parrot commits "errors of understanding" because nobody would give them billions if they spoke accurately and said "our system is schizophrenic, delusional, and psychotic."
This is the sort of thing I say in response to "hallucination" claims (no idea whether the post you're thinking of came from me, but I was commenting here around that time). Here's the longer version:
There is nothing in the process of generating "high probability" strings of text that involves the truth value of what the text means (as interpreted by a human being). Meaning is just irrelevant from the LLM's perspective.
Where people get confused, I think, is in squaring this with the fact that LLMs produce "true" statements far more often than they produce "false" statements. If LLMs have no understanding of semantic meaning and no concept of truth, then how is it that they consistently write things that are sensible and true?
My answer is that "high probability" strings of text will also tend to be sensible and true strings of text, given how the LLMs were trained. I know people like to complain that the internet is full of nonsense, but *most* of what's out there is sensible and true. If I ask an LLM a question that it's seen correctly answered thousands of times, then a statistically plausible string of generated text will likely be a true string of generated text. And this is the case despite the fact that the LLM has no concept of truth.
Well, as it turns out, sometimes a statistically plausible string of text will be nonsensical, or false. Us humans label these as "hallucinations", because we understand truth from falsehood. But, from the LLM's perspective, it's just cranking out text that looks like the sort of text it's seen - hence it is doing the same thing when it "hallucinates" as when it tells the truth.
Karpathy also has a quote about how "hallucination is all LLMs do" and pointing out that to get reliability from a system built on LLMs, we need to add additional layers of analysis / methods - https://twitter.com/karpathy/status/1733299213503787018
"However, the underlying cause of all hallucinations, at least in large language models, is that the current language modeling paradigm used in these systems is, by design, a hallucination machine."
The architecture of the Transformer makes hallucinations inevitable given the combination of a large model, large training set, and extended training period. The heuristic reasons for this claim are in a paper by B.A. Huberman and me: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4676180
Gary, could you possibly speculate about what such new ideas might be?
And do you think they are going to be kind of "fixes" for the existing LLM-first strategy (e.g. incorporating non-LLM components into a workflow to decrease hallucinations, like it is now done with custom GPTs and third-party software), or do you think LLMs are fundamentally a dead end on the AI progress path, and nothing more than just a brench of evolution, and not the most successful one?
Hi Andrii, what I have advised to startups working on a solution that started with an LLM, is to add word pre-processors and post-processors to tailor the input to the LLM and its output to more accurately respond to a prompt.
One common problem that I found when Alpha-testing an LLM, in a specialized knowledge domain, is that it lacked the ability to recognize synonyms. I proposed a synonym pre-processor, it was implemented, and now the software provides measurably better results.
Likewise, post-processors allow for small corrections to the LLM’s output; these corrections can be capitalization.
I didn't say it would be easy, I'm just advocating the development of advanced AI/AGI in the best interests of all mankind, rather than some subset. Also, the closer you get to superintelligence, the more dangerous "Fast experimentation and failure" becomes.
You do realise that, in the context of superintelligent agentic AGI, "all that [an open-ended race] entails" includes the possibility of massive societal harm at global scale, up to and including the possibility of human extinction...? Isn't a different approach thus justified...?
We shouldn't perhaps even call them hallucinations - for one can hallucinate against the expectation of normative cognitive behaviour, which is clearly something LLMs don't possess (even if they often succeed in mimicking)
What is your point? Since we humans exaggerate or drift of course that it is Okay for AIs to do so? With Humans we often have vigorous critiques, discussions, alternative views expressed openly. Do AIs cross check and challenge each other.?
Human intelligence gets things wrong and hallucinates! Actually, some improvements in scientific understanding likely start as something reassembling hallucination (Special/General Theory of Relativity?). Intelligence itself innately will have flaws and be incorrect sometimes just like the very smartest and considerate/deliberate human's reasoning/response. The fact is that these LLMs and the neural networks in general are sorta doing magic stuff like our brains do. Brains are often wrong... A perfect intelligence, artificial or otherwise is almost certainly impossible. I except flaws from AI agents persisting into the future. Those flaws may become more rare and more subtle but I don't think they will ever go away completely.
Have you read Einstein's own exposition on how he came to the ideas of special relativity? It's quite lucid. It seems almost inevitable as you follow the line of reasoning inexorably toward its conclusion. It doesn't seem hallucinatory in the least, only novel. Language model "hallucinations" seem neither inevitable nor novel; there are millions of ways to make up a plausible sounding title for a fake citation.
IF LLMs cannot recognize the difference between statistically accurate and hallucinated responses then how can this problem ever be corrected? Only human outsiders can understand this distinction.
Statistical Accuracy requires Exclusive Middle Sets and what it has been told and that's not the way Things Work. We went through this in the 80s with Expert Systems and in the last decade with Watson.
To avoid acknowledging that human consciousness is a fundamentally hallucinatory process, and that even the best and brightest fall victim to it *without realizing it* regularly?
Personally, my strong anti-war sentiments seem to compel me to forcefully inject this fact into conversations, in hopes people can learn that "the reality" is an illusion.
"new" approaches based on "different" ideas - exactly what I propose!
Intelligence is the ability to handle differences. "the same approaches" will not work, but what is meaningfully different? With respect to any result what actions are different? Even if two objects or two actions are different when can they be interchangeable (generalizable)?
Goals - states different from the current one. Plans - actions that will change our state according to our goals. Differences are everywhere. They are the core concept to understand intelligence.
Another person commented on this substack a year ago, LLMs are doing the same thing when they get it right that they are doing when they get it wrong. They said it better, but I’ve never forgotten.
I like that way of putting it. If you come up with the better version, please let us know here in a comment.
It might have been me... (see other comment)
I had the original in my notes - was this yours? “When an LLM ‘hallucinates’, it’s doing the exact same thing it does when it tells the truth.” Would have been from September 2023 or shortly before.
Could very well be. I've said it many times. But many people, including Gary have said the same as it is rather factual. From https://ea.rna.nl/2023/11/01/the-hidden-meaning-of-the-errors-of-chatgpt-and-friends/ (that one is from september last year, when I was preparing my talk):
"Which means that showing all these errors is fine, but labelling them ‘errors of understanding’ may have the opposite effect of what we realists want to achieve, as the underlying message is interpreted as: “the (real) understanding is there, it just goes wrong sometimes“. In reality there is no understanding to begin with, so that hidden, unintended, message is wrong.
The system calculates, and the ‘error’ is not an ‘error’ from the perspective of that calculation. What we see as an error is what the system has calculated as ‘the best’. It is ‘correct’. Exactly the same way as it has calculated the results we humans consider correct. We should therefore not say ‘the system (still) makes error of understanding’ but we should say ‘the system lacks understanding, so incorrect results are to be expected’. Do not talk about ‘errors’."
Another one that is one thing producing both the wanted and unwanted effect is 'memorisation' which can be something we like (GPT correctly quotes Shakespeare) and dislike (GPT correctly quotes copyrighted material — i.e. effectively data leakage)
Another one: "It is also good to repeat another perspective here. GenAI doesn’t produce ‘results’ and ‘hallucinations’. It produces ‘approximations’, period. Even its correct answers are approximations of understanding, not understanding itself." from https://ea.rna.nl/2024/02/13/will-sam-altmans-7-trillion-ai-plan-rescue-ai/ That one also contains some rough calculations about scaling (and how that is not going to solve the problem). Gary has also mentioned that both correct and incorrect results are both (good and bad) approximations: https://garymarcus.substack.com/p/sora-cant-handle-the-truth
Personally appreciate the approximations framing as well because it gets how genAI may be an additional layer distancing us from the pursuit of truth goal, rather than the promised ability to get to truth faster.
Science is also an approximation of understanding, not understanding itself. I don't think that framing is helpful. What y'all are trying to get at is some lack of deeper model, not the degree of truthfulness per se.
AI Numerologists have to say their word salad spewing stochastic parrot commits "errors of understanding" because nobody would give them billions if they spoke accurately and said "our system is schizophrenic, delusional, and psychotic."
So how do you put it?
This is the sort of thing I say in response to "hallucination" claims (no idea whether the post you're thinking of came from me, but I was commenting here around that time). Here's the longer version:
There is nothing in the process of generating "high probability" strings of text that involves the truth value of what the text means (as interpreted by a human being). Meaning is just irrelevant from the LLM's perspective.
Where people get confused, I think, is in squaring this with the fact that LLMs produce "true" statements far more often than they produce "false" statements. If LLMs have no understanding of semantic meaning and no concept of truth, then how is it that they consistently write things that are sensible and true?
My answer is that "high probability" strings of text will also tend to be sensible and true strings of text, given how the LLMs were trained. I know people like to complain that the internet is full of nonsense, but *most* of what's out there is sensible and true. If I ask an LLM a question that it's seen correctly answered thousands of times, then a statistically plausible string of generated text will likely be a true string of generated text. And this is the case despite the fact that the LLM has no concept of truth.
Well, as it turns out, sometimes a statistically plausible string of text will be nonsensical, or false. Us humans label these as "hallucinations", because we understand truth from falsehood. But, from the LLM's perspective, it's just cranking out text that looks like the sort of text it's seen - hence it is doing the same thing when it "hallucinates" as when it tells the truth.
Karpathy also has a quote about how "hallucination is all LLMs do" and pointing out that to get reliability from a system built on LLMs, we need to add additional layers of analysis / methods - https://twitter.com/karpathy/status/1733299213503787018
Good point. Truth and statistics are different animals. If you want a cat, don't get a dog.
I like how Alejandro Piad Morffis states it here:
"However, the underlying cause of all hallucinations, at least in large language models, is that the current language modeling paradigm used in these systems is, by design, a hallucination machine."
https://blog.apiad.net/p/reliable-ai-is-harder-than-you-think
Our language also seems designed for hallucination...whether this is by design I cannot say, but I am very suspicious. 😐
Hmm, maybe hallucinations are absolutely core to how human consciousness works. -- Jess
you got to this 40 minutes before I did... I just wrote more!
The architecture of the Transformer makes hallucinations inevitable given the combination of a large model, large training set, and extended training period. The heuristic reasons for this claim are in a paper by B.A. Huberman and me: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4676180
Gary, could you possibly speculate about what such new ideas might be?
And do you think they are going to be kind of "fixes" for the existing LLM-first strategy (e.g. incorporating non-LLM components into a workflow to decrease hallucinations, like it is now done with custom GPTs and third-party software), or do you think LLMs are fundamentally a dead end on the AI progress path, and nothing more than just a brench of evolution, and not the most successful one?
Hi Andrii, what I have advised to startups working on a solution that started with an LLM, is to add word pre-processors and post-processors to tailor the input to the LLM and its output to more accurately respond to a prompt.
One common problem that I found when Alpha-testing an LLM, in a specialized knowledge domain, is that it lacked the ability to recognize synonyms. I proposed a synonym pre-processor, it was implemented, and now the software provides measurably better results.
Likewise, post-processors allow for small corrections to the LLM’s output; these corrections can be capitalization.
Cue shameless self-promotion! :-) -- https://www.bigmother.ai
I didn't say it would be easy, I'm just advocating the development of advanced AI/AGI in the best interests of all mankind, rather than some subset. Also, the closer you get to superintelligence, the more dangerous "Fast experimentation and failure" becomes.
You do realise that, in the context of superintelligent agentic AGI, "all that [an open-ended race] entails" includes the possibility of massive societal harm at global scale, up to and including the possibility of human extinction...? Isn't a different approach thus justified...?
We shouldn't perhaps even call them hallucinations - for one can hallucinate against the expectation of normative cognitive behaviour, which is clearly something LLMs don't possess (even if they often succeed in mimicking)
What is your point? Since we humans exaggerate or drift of course that it is Okay for AIs to do so? With Humans we often have vigorous critiques, discussions, alternative views expressed openly. Do AIs cross check and challenge each other.?
Human intelligence gets things wrong and hallucinates! Actually, some improvements in scientific understanding likely start as something reassembling hallucination (Special/General Theory of Relativity?). Intelligence itself innately will have flaws and be incorrect sometimes just like the very smartest and considerate/deliberate human's reasoning/response. The fact is that these LLMs and the neural networks in general are sorta doing magic stuff like our brains do. Brains are often wrong... A perfect intelligence, artificial or otherwise is almost certainly impossible. I except flaws from AI agents persisting into the future. Those flaws may become more rare and more subtle but I don't think they will ever go away completely.
Have you read Einstein's own exposition on how he came to the ideas of special relativity? It's quite lucid. It seems almost inevitable as you follow the line of reasoning inexorably toward its conclusion. It doesn't seem hallucinatory in the least, only novel. Language model "hallucinations" seem neither inevitable nor novel; there are millions of ways to make up a plausible sounding title for a fake citation.
IF LLMs cannot recognize the difference between statistically accurate and hallucinated responses then how can this problem ever be corrected? Only human outsiders can understand this distinction.
Statistical Accuracy requires Exclusive Middle Sets and what it has been told and that's not the way Things Work. We went through this in the 80s with Expert Systems and in the last decade with Watson.
Can you name a single human *that has public writing on the internet* (so I can check their work) that does not hallucinate?
I would bring up the distinction here between lying and bullshitting (in Frankfurt's meaning).
To avoid acknowledging that human consciousness is a fundamentally hallucinatory process, and that even the best and brightest fall victim to it *without realizing it* regularly?
Personally, my strong anti-war sentiments seem to compel me to forcefully inject this fact into conversations, in hopes people can learn that "the reality" is an illusion.
Even Sam Altman has publicly stated that they're not *bugs*, they're *features*. https://ea.rna.nl/2023/11/01/the-hidden-meaning-of-the-errors-of-chatgpt-and-friends/
Interesting no one has yet mentioned Karl Friston and active inference as a new approach or new idea!
Although key here is rather Pierce's abductive reasoning, which resists computation and formalisation.
Xu, Z.; Jain, S.; and Kankanhalli, M. 2024. Hallucination is Inevitable: An Innate Limitation of Large
Language Models. https://arxiv.org/abs/2401.11817
"new" approaches based on "different" ideas - exactly what I propose!
Intelligence is the ability to handle differences. "the same approaches" will not work, but what is meaningfully different? With respect to any result what actions are different? Even if two objects or two actions are different when can they be interchangeable (generalizable)?
Goals - states different from the current one. Plans - actions that will change our state according to our goals. Differences are everywhere. They are the core concept to understand intelligence.
Wait, what!? Someone told me this was the AI’s way of being creative…