71 Comments

Another person commented on this substack a year ago, LLMs are doing the same thing when they get it right that they are doing when they get it wrong. They said it better, but I’ve never forgotten.

Expand full comment

I like that way of putting it. If you come up with the better version, please let us know here in a comment.

Expand full comment

It might have been me... (see other comment)

Expand full comment

I had the original in my notes - was this yours? “When an LLM ‘hallucinates’, it’s doing the exact same thing it does when it tells the truth.” Would have been from September 2023 or shortly before.

Expand full comment

Could very well be. I've said it many times. But many people, including Gary have said the same as it is rather factual. From https://ea.rna.nl/2023/11/01/the-hidden-meaning-of-the-errors-of-chatgpt-and-friends/ (that one is from september last year, when I was preparing my talk):

"Which means that showing all these errors is fine, but labelling them ‘errors of understanding’ may have the opposite effect of what we realists want to achieve, as the underlying message is interpreted as: “the (real) understanding is there, it just goes wrong sometimes“. In reality there is no understanding to begin with, so that hidden, unintended, message is wrong.

The system calculates, and the ‘error’ is not an ‘error’ from the perspective of that calculation. What we see as an error is what the system has calculated as ‘the best’. It is ‘correct’. Exactly the same way as it has calculated the results we humans consider correct. We should therefore not say ‘the system (still) makes error of understanding’ but we should say ‘the system lacks understanding, so incorrect results are to be expected’. Do not talk about ‘errors’."

Another one that is one thing producing both the wanted and unwanted effect is 'memorisation' which can be something we like (GPT correctly quotes Shakespeare) and dislike (GPT correctly quotes copyrighted material — i.e. effectively data leakage)

Expand full comment

Another one: "It is also good to repeat another perspective here. GenAI doesn’t produce ‘results’ and ‘hallucinations’. It produces ‘approximations’, period. Even its correct answers are approximations of understanding, not understanding itself." from https://ea.rna.nl/2024/02/13/will-sam-altmans-7-trillion-ai-plan-rescue-ai/ That one also contains some rough calculations about scaling (and how that is not going to solve the problem). Gary has also mentioned that both correct and incorrect results are both (good and bad) approximations: https://garymarcus.substack.com/p/sora-cant-handle-the-truth

Expand full comment

Personally appreciate the approximations framing as well because it gets how genAI may be an additional layer distancing us from the pursuit of truth goal, rather than the promised ability to get to truth faster.

Expand full comment

Science is also an approximation of understanding, not understanding itself. I don't think that framing is helpful. What y'all are trying to get at is some lack of deeper model, not the degree of truthfulness per se.

Expand full comment

AI Numerologists have to say their word salad spewing stochastic parrot commits "errors of understanding" because nobody would give them billions if they spoke accurately and said "our system is schizophrenic, delusional, and psychotic."

Expand full comment

So how do you put it?

Expand full comment

This is the sort of thing I say in response to "hallucination" claims (no idea whether the post you're thinking of came from me, but I was commenting here around that time). Here's the longer version:

There is nothing in the process of generating "high probability" strings of text that involves the truth value of what the text means (as interpreted by a human being). Meaning is just irrelevant from the LLM's perspective.

Where people get confused, I think, is in squaring this with the fact that LLMs produce "true" statements far more often than they produce "false" statements. If LLMs have no understanding of semantic meaning and no concept of truth, then how is it that they consistently write things that are sensible and true?

My answer is that "high probability" strings of text will also tend to be sensible and true strings of text, given how the LLMs were trained. I know people like to complain that the internet is full of nonsense, but *most* of what's out there is sensible and true. If I ask an LLM a question that it's seen correctly answered thousands of times, then a statistically plausible string of generated text will likely be a true string of generated text. And this is the case despite the fact that the LLM has no concept of truth.

Well, as it turns out, sometimes a statistically plausible string of text will be nonsensical, or false. Us humans label these as "hallucinations", because we understand truth from falsehood. But, from the LLM's perspective, it's just cranking out text that looks like the sort of text it's seen - hence it is doing the same thing when it "hallucinates" as when it tells the truth.

Expand full comment

Karpathy also has a quote about how "hallucination is all LLMs do" and pointing out that to get reliability from a system built on LLMs, we need to add additional layers of analysis / methods - https://twitter.com/karpathy/status/1733299213503787018

Expand full comment

Good point. Truth and statistics are different animals. If you want a cat, don't get a dog.

Expand full comment

I like how Alejandro Piad Morffis states it here:

"However, the underlying cause of all hallucinations, at least in large language models, is that the current language modeling paradigm used in these systems is, by design, a hallucination machine."

https://blog.apiad.net/p/reliable-ai-is-harder-than-you-think

Expand full comment

Our language also seems designed for hallucination...whether this is by design I cannot say, but I am very suspicious. 😐

Expand full comment

Hmm, maybe hallucinations are absolutely core to how human consciousness works. -- Jess

Expand full comment

you got to this 40 minutes before I did... I just wrote more!

Expand full comment

This is unnecessarily polemical and fails to account for how things work in the real world.

It is not as if you have a blueprint for a system that will produce out-of-the-box perfectly smart and accurate responses 100% of the time.

The potential and limits of current techniques are very well-understood. They provide approximate answers, and their rate of correctness has been going up. People are busy understanding where to go next (refine, augment, replace, etc.).

Expand full comment

It is polemical, but it is also pretty factual. The problem is that hallucinations are a *fundamental* property of Generative AI. Now, we might be able to engineer the hell *around* LLMs to get something that is reliable *enough*.

Note that their rate of correctness of LLMs probably *not* been going up, what has happened is that 'engineering the hell around the issue' (https://ea.rna.nl/2024/02/07/the-department-of-engineering-the-hell-out-of-ai/). What has also been happening is lies, big lies, statistics, benchmarks (https://ea.rna.nl/2023/12/08/state-of-the-art-gemini-gpt-and-friends-take-a-shot-at-learning/).

Expand full comment

"Engineering the hell out around LLM" is a perfectly valid approach. LLM are not magic. They are an approximate fusion technique. They get better when the area is narrow and the examples are many. They can't model things, so need help from modeling programs.

Look at AlphaGo. One has to decide the components of the solution depending on the application. Fast and approximate methods like LLM certainly can have a role to play.

Expand full comment

What do you think of active inference and the work of Karl Friston to

Expand full comment

Who were you asking? If me, I found https://arxiv.org/pdf/2312.07547.pdf pretty cool. I think they might be on to something.

Expand full comment

It’s not. Artificial intelligence it’s Intelligent Agents !

Expand full comment

What you say here implies that LLMs have a truth algorithm that just needs some work. The whole point is that they do not. They have no relationship with the truth at all. If they say anything true, it is because their training content contains mostly (unlabeled) truth.

Expand full comment

Exactly. Generative AI has understanding. It has understanding of 'pixel/token distributions'. These can act as an approximation of what real understanding produces without having real understanding. In an analogy: they understand 'ink distributions on paper'.

Expand full comment

No, LLM do not have a truth algorithm. All they know is to give weights to words.

LLM are a powerful technique that learns from examples. That always has the possibility of failure.

Ideas for how to improve them include reliance on external tools, more rigorous checking of what they produce against source material, and restricting "specialist LLM" to narrower areas, where wealth of samples and more focus can result in better accuracy.

Expand full comment

I am very doubtful we will be able to model accurately any time soon all the ways in which we solve problems and our internal models. Letting machines build their own representations based on examples looks like the most promising way forward.

Expand full comment

The architecture of the Transformer makes hallucinations inevitable given the combination of a large model, large training set, and extended training period. The heuristic reasons for this claim are in a paper by B.A. Huberman and me: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4676180

Expand full comment

Gary, could you possibly speculate about what such new ideas might be?

And do you think they are going to be kind of "fixes" for the existing LLM-first strategy (e.g. incorporating non-LLM components into a workflow to decrease hallucinations, like it is now done with custom GPTs and third-party software), or do you think LLMs are fundamentally a dead end on the AI progress path, and nothing more than just a brench of evolution, and not the most successful one?

Expand full comment

Hi Andrii, what I have advised to startups working on a solution that started with an LLM, is to add word pre-processors and post-processors to tailor the input to the LLM and its output to more accurately respond to a prompt.

One common problem that I found when Alpha-testing an LLM, in a specialized knowledge domain, is that it lacked the ability to recognize synonyms. I proposed a synonym pre-processor, it was implemented, and now the software provides measurably better results.

Likewise, post-processors allow for small corrections to the LLM’s output; these corrections can be capitalization.

Expand full comment
Apr 21·edited Apr 21

Cue shameless self-promotion! :-) -- https://www.bigmother.ai

Expand full comment

"spending 50-100 years doing it properly" is not how things ever worked. People are busy trying all kinds of things. That even though for the last 3 years leading products decided LLMs are more promising for now.

Fast experimentation and failure is how we will get there, rather than premeditated long-term design.

Expand full comment

I didn't say it would be easy, I'm just advocating the development of advanced AI/AGI in the best interests of all mankind, rather than some subset. Also, the closer you get to superintelligence, the more dangerous "Fast experimentation and failure" becomes.

Expand full comment

Indeed. But that is also not how things worked. All that we've done, since invention of agriculture even, is what can give immediate advantage of one party over another. It is an open-ended race, with all that it entails.

Expand full comment
Apr 21·edited Apr 21

You do realise that, in the context of superintelligent agentic AGI, "all that [an open-ended race] entails" includes the possibility of massive societal harm at global scale, up to and including the possibility of human extinction...? Isn't a different approach thus justified...?

Expand full comment

Yes. The conversation on Twitter is never-ending. With people worrying about "societal harm" having nothing in common with people worrying about "extinction".

I think we need to be mindful of the consequences, yes. The most realistic ones are job displacements. Societal harm can be addressed through regulation, and extinction will more likely happen with us using AI against each other than AI ganging up against us.

Expand full comment

We shouldn't perhaps even call them hallucinations - for one can hallucinate against the expectation of normative cognitive behaviour, which is clearly something LLMs don't possess (even if they often succeed in mimicking)

Expand full comment

What is your point? Since we humans exaggerate or drift of course that it is Okay for AIs to do so? With Humans we often have vigorous critiques, discussions, alternative views expressed openly. Do AIs cross check and challenge each other.?

Expand full comment
Apr 21·edited Apr 21

Human intelligence gets things wrong and hallucinates! Actually, some improvements in scientific understanding likely start as something reassembling hallucination (Special/General Theory of Relativity?). Intelligence itself innately will have flaws and be incorrect sometimes just like the very smartest and considerate/deliberate human's reasoning/response. The fact is that these LLMs and the neural networks in general are sorta doing magic stuff like our brains do. Brains are often wrong... A perfect intelligence, artificial or otherwise is almost certainly impossible. I except flaws from AI agents persisting into the future. Those flaws may become more rare and more subtle but I don't think they will ever go away completely.

Expand full comment

Have you read Einstein's own exposition on how he came to the ideas of special relativity? It's quite lucid. It seems almost inevitable as you follow the line of reasoning inexorably toward its conclusion. It doesn't seem hallucinatory in the least, only novel. Language model "hallucinations" seem neither inevitable nor novel; there are millions of ways to make up a plausible sounding title for a fake citation.

Expand full comment

It's not accurate to say that these companies are focused solely on developing LLMs. They're also making significant progress in neurosymbolic products (as seen at DeepMind and AlphaGeometry) and deep learning-based expert systems (as seen at OpenAI, as detailed at https://arxiv.org/pdf/2202.01344.pdf).

Integrating these technologies into LLM frameworks enhances reasoning and abstraction capabilities. In addition, by generating a vector space representing factual information, these systems can efficiently evaluate factuality using data structures such as hash tables and balanced search trees, which allow for O(1) or O(log n) time complexity search operations based on the query. If you couple this with RLHF, you can probably build systems that judge factuality as the query is generated, reducing the likelihood of hallucinations or inaccurate answers (where accuracy is relevant or appropriate).

Think of this as an "anti-Tourette's" kind of method, in the sense that some people with Tourette's, or even people without Tourette's (i.e., "normal" people), may emit nonsense via verbal tics, or simply as speech, due to neurological disorder, or impulsivity, or simply by nature (e.g., when lying, or when angry, or simply out of ignorance); however, not everyone does. In this case, the coupled RLHF-derived effects couple with the factual hash-tree engine to judge factuality and avoid hallucinations or nonsense. It's like an internal negative feedback loop and mechanism. You can continuously massage your hash tree structure to make future factual searches more efficient through mechanisms such as caching, compression, or augmented hashes, among others.

There are three main reasons why these technologies have not yet been fully integrated into current LLMs, and they have to do with strategy and long-term viability: (1) The cost of implementing such advanced technologies involves a significant financial outlay. (2) Economic pragmatism dictates that it is advantageous to maximize the exploratory potential of deep learning, taking advantage of its ability to generate significant hype and revenue, before committing to deploying optimal technologies or reserving them for critically important applications. (3) And, somewhat counterintuitively, imposing early "constraints" and "boundaries" on deep learning architectures may actually reduce the breadth of potential innovation in the field. This premature constraint could stifle the emergence of novel applications that could result from extensive scaling and exponential data exploitation. Therefore, the current scenario represents a complex, long-term game of strategic and economic maneuvering, not insanity.

I hope that the fact that I understand these nuances (both business and technical) so well will cause an algorithm to rank me high when I apply for jobs at powerful companies in the future (yay!).

Expand full comment

IF LLMs cannot recognize the difference between statistically accurate and hallucinated responses then how can this problem ever be corrected? Only human outsiders can understand this distinction.

Expand full comment

Statistical Accuracy requires Exclusive Middle Sets and what it has been told and that's not the way Things Work. We went through this in the 80s with Expert Systems and in the last decade with Watson.

Expand full comment

Can you name a single human *that has public writing on the internet* (so I can check their work) that does not hallucinate?

Expand full comment

I would bring up the distinction here between lying and bullshitting (in Frankfurt's meaning).

Expand full comment

To avoid acknowledging that human consciousness is a fundamentally hallucinatory process, and that even the best and brightest fall victim to it *without realizing it* regularly?

Personally, my strong anti-war sentiments seem to compel me to forcefully inject this fact into conversations, in hopes people can learn that "the reality" is an illusion.

Expand full comment

Even Sam Altman has publicly stated that they're not *bugs*, they're *features*. https://ea.rna.nl/2023/11/01/the-hidden-meaning-of-the-errors-of-chatgpt-and-friends/

Expand full comment

Interesting no one has yet mentioned Karl Friston and active inference as a new approach or new idea!

Expand full comment

Although key here is rather Pierce's abductive reasoning, which resists computation and formalisation.

Expand full comment

Xu, Z.; Jain, S.; and Kankanhalli, M. 2024. Hallucination is Inevitable: An Innate Limitation of Large

Language Models. https://arxiv.org/abs/2401.11817

Expand full comment