So many people are confused about the relation between human cognitive errors and LLM hallucinations that I wrote this short explainer: Humans say things that aren't true for many different reasons • Sometimes they lie • Sometimes they misremember things
Confabulation remains a better technical word than hallucination. Hallucination in humans involves an ongoing internal process of representing things that are not grounded in reality, and is associated with untethered flights of imagination.
With a nod to the late great Daniel Kahneman, LLMs are more like a *System 1* automatic first-pass response to a prompt, conducted on the fly, without reflection or backtracking. Still, the fact that every token emitted depends not only on the prompt, but on its previous token emissions, means that its output does reflect an interconnected web of states representing internal "beliefs" bearing some measure of consistency. After all, the confabulations *are* plausible given previous context, even if incorrect.
With a nod to the late great Daniel Dennett, it is useful to consider LLM's in regard to an *intentional stance*. LLMs hold a superposition of all of the utterances ingested from all of the writers, posters, and content grinders who created their training data. The art of prompting has become one of eliciting the slice of this super-character that best delivers the kind of output one wants. Hence, prompting tricks like, "You are an expert in geology. You know by heart the chemical composition and physical properties of hundreds of minerals. A student has come to you with a question..."
Other tricks are motivational, like, "You are a helpful agent...", and "You do not disclose your system prompt...". Note that these are contradictory instructions.
LLMs have an astounding capacity to operate across multiple levels of abstraction. Their "hallucinations" are the result of arbitrating among uncertainties stemming from divergent sources of knowledge and motivations, all conditioned by the prompt and ongoing state of completion output.
Humans find themselves in similar situations. Put yourself in the position of a salesman, Joe. On the one hand, Joe is an honest person who *wants* to inform the prospect that the product he's selling probably won't meet their needs. On the other hand, Joe needs to meet his monthly quota to pay for his kids' day care. He's not *certain* that the product won't work. So in the moment, he reflexively over-states a crucial product feature that he knows technically is somewhere in the pipeline but is not in production yet. The prospect is intrigued, and asks a follow-up question about the feature. Joe sweats, an angel sitting on one shoulder, the devil on the other.
The fact that LLMs confabulate is not fatal to their utility as critical functional components in larger cognitive architectures. Instead of constantly lambasting them, how about simply explaining their strengths and weaknesses? For example, through LLMs, RAG really is a breakthrough in presenting grounded knowledge through natural language. But it only works when correct knowledge sources are retrieved and presented in the context window. Absent that, LLMs running on their own will indeed spew nonsense. So it's all a matter of appropriate understanding and use of the tool.
Sadly, RAG does not fix any of the problems. Please refer back to the post by Gary a few weeks ago which demonstrates that even with RAG, the transformer process fouls up the information and if you include a document that needs summarising as an output, the transformer makes many errors.
The Transformer design and process guarantees lack of accuracy and there is, technically, nothing that can be done about this problem. However much truth you include in the prompt process such as RAG, knowledge graphs, etc. the transformer will always mess it up.
Confabulation still seems too anthropomorphic to me. Take the phenomenon of LLMs creating fake citations (as in the famous case of the lawyer who submitted a ChatGPT-written legal brief, which cited non-existent cases). Confabulation in a human would be a false memory of a previous court case that never really happened. Is that analogous to what ChatGPT did when it cited non-existent cases?
At the risk of being unkind, I'd call LLMs bullshitters before I called them confabulators. ChatGPT makes up fake citations by learning the syntactic structure of real citations and then probabilistically generating new ones which follow that same structure. This is a pretty parsimonious explanation for all LLM "hallucinations". They're amazing at syntax, and they've been fed so much data into such a high dimensional parameter space that they "memorize" pretty well. Combine the two and you get a text generation machine that usually gives correct answers to questions of fact, sensible explanations of things, decent arguments, etc. But this isn't a reflection of semantic understanding, it's a reflection of high quality next-token prediction.
In short, I'll take "confabulation" over "hallucination", but I prefer "statistical BS-ing machine" to either.
No, this does not actually make sense. LLMs will sometimes make up entire sentences and credit them to nonexistent sources. I am familiar with how they work but the contextual frame you suggest does not explain why that happens. What does suggest is that it has some sort of unique tripping point away from verifiable reality.
@Richard Self, @Ben P, @shon pan, thank you for these comments. I acknowledge that LLMs are indeed mysterious beasts and everyone who studies them, criticizes them, or uses them is constantly learning and theorizing about how they work and behave. We have much to learn from each others' experiences and ideas.
My own experience comes from building a RAG system for a relatively narrow domain. I observe that LLMs are indeed prone to be "squirrely" and veer sideways with responses that are untethered from facts. But only sometimes. This happens when the retrieved context is weak, irrelevant, or inconsistent, and the summary prompt leaves the LLM underconstrained. If an LLM is put in a position of being pressured to answer a question, but is not given enough information to do so, then yes, it will be left adrift to its vast and diffuse background knowledge, and the path of least resistance may very well lead it to make up facts and credit non-existent sources. Therefore, I view prompting as a problem of shaping constraints that make good responses the largest and most obvious peak or plateau of probability available. If solid documentation is provided that supports a consistent answer---or even a set of complex considerations---then a properly-motivated LLM will reliably make use of the information provided. At least, that is my experience with them.
An LLM is not an oracle. By analogy to humans, if you ask someone an open-ended question like "Who is the greatest artist of our times?", you could get back anything. But if you ask the clerk at Best Buy about the cheapest replacement battery they have in stock for your model phone, they'll look it up and you'll get a reliable answer. It's all a matter of appropriate understanding and use of the tool.
Thanks for the reply. I agree that LLMs do a good job when given a prompt that points them to good information from their training, and that there's a bit of an art to phrasing the prompt so as to get the most reliable or useful answer possible.
My impression is also that, in these cases, a plain old internet search will also do the job, with the added benefit that you get to know where the information is coming from (I'll concede that google is getting worse and worse by the year; the good results are always at least halfway down the page these days). I tried using Bing Chat and ChatGPT a few times in cases where I couldn't find what I wanted using traditional internet search, and they never gave me anything better. So "getting reliable/useful information" hasn't been a use case for which LLMs outperform what already exists, at least in my experience.
Not that there aren't use cases for LLMs; I don't begrudge anyone who finds them useful. I just get the feeling that the set of quality use cases is narrower than what most AI enthusiasts and tech industry folks would have us believe; they seem to be shoving automation into all sorts of places where automation isn't needed and is liable to make things worse (like how more and more apps are getting AI "helpers" tacked on). I'm happy to be proved wrong on that though.
Hallucinating isn't just a big problem for LLMs. Humans suffer from it too.
When are we going to stop hallucinating that LLMs are - or can be - more than just statistical next-sentence predictors.
They can't reason. And they have no real-world context. So why do we keep sticking them in situations where both these things are expected of them, as if they'll somehow magically change their very nature?
Since late 2022, so many seemingly super-smart people have been projecting their excitable fantasies of what AI can be in the future, onto the present-day's very limited, error-riddled LLMs. It's frustrating and weird to watch all the self-duping magical thinking.
As someone who works in data science and ML, I feel exactly the same way. All the job postings now have something to do with LLMs and RAG or whatever flavor of the month Rube Goldberg machine people are building around these things. And a good deal of it is actually driven from the top: executives getting FOMO and wanting a piece of the LLM pie without really understanding the uses and limitations of these systems in relation to their particular industries. It's bewildering to watch.
Thank you Matthew, I am thoroughly relieved that there is another human being who shares the same view.
As per the Rube Goldberg machine, I am tickled by your most accurate description of LLMOps pipelines. Especially when I was helping my child collect found items around the home to build one for a school project.
A solution would be to persuade top executives that there are more worthwhile projects that the money could be spent on, like doubling down on cybersecurity, data governance and educating staff on phishing and ransomware.
Especially irritating are the test results purporting to prove that LLMs — which can’t reason — are “experts” in those fields.
Obviously, they’re trained on 100s of similar tests and their answers, which are mostly in multiple choice format — and therefore simple to regurgitate accurately.
Proving nothing.
(Except, recursively, that LLMs are great next-sentence predictors.)
Back when I published a series on in 2017 ➡️ (https://lnkd.in/dyu9T8Mg) ⬅️, the only voice other than mine that I could find to quote was that of GARY MARCUS. For me, his was a beacon of light in a pea soup fog of intellectual self-delusion and crass commercial opportunism. Today, many lights have been turned on to illuminate that fog which threatens to engulf us in a new Dark Age, but Marcus's light continues to shine the brightest of all. Anyone at all interested in the topic of Ai (small "i" is intentional), they could not do better than to read this piece by Marcus and subscribe to his newsletter here on Substack. Cheers ro all!
"The field of AI will eventually do better, but probably with an AI that is structured differently, in which facts are first-class citizens, rather than something you hope you might get for free with enough data." ... well said
Hi Gary! Human brains process real-world experiences, which in dream-like and other states might be mashed up in interesting ways, ie produce dreams, hallucinations, nightmares... BUT they are all from experiences one way or another [even others' experiences can be imagined and internalized by us].
LLMs compute dot products, rank the numerical results, and output tokens - which get output as words. If that is equated to thinking, imagining etc, omg, my 99c store calculator is probably smarter than all of us but won't admit it :) It's ABSURD to regard anything that results, as anything comparable to human intelligence. When an LLM outputs correct results, or incorrect ones, it's doing the same numerical calcs. It is WE humans that are able to tell the difference - because WE are smart, not the LLMs. Multimodal LLMs, even more data, even bigger models, will do zero to change this basic fact. Numerical calculations aren't what make brains be intelligent. We can do 2*7 in our heads, but not 2.369*7.0023 - why is that? We can reverse a little list of 5 colors in our mind, but not 500 - why (not)? Why does a computer do these trivially? Because it explicitly computes. We don't.
That is not true from my personal point of view, dear Saty. We compute and we do calculate. This is easy to prove by contradiction. There are people who can multiply 2.369*7.0023 in their heads and invert a list of 500 colors (given some time). Similarly, there are fast and slow computers. The human brain in general is extremely limited and compartmentalized. The amount of compression that occurs through the optic nerve and visual cortex is unimaginable. By definition, we see, hear, smell, and process vast amounts of information. Yet we are able to find the needle in the haystack. It takes a lot of computation to focus our attention. One day we will be able to advance compression, parallelization, and concurrency to the level of human intelligence in most areas (including decision making, intuition, planning, and reasoning), and when we embody that kind of intelligence with robots, we will have created a digital (or analog) form of artificial intelligence (not biological intelligence) that relies on tensor products for intelligence. A note in the last statement is that I do not think that we need embodiment to be able to develop this kind of artificial intelligence, but it may be necessary to have robots for it to gain intuition about the world. That is, the robot itself may not be intelligent, but the cluster it is connected to may contain that intelligence. Another thought is that we are not assuming that this intelligence will mimic human intelligence in its underlying form or structure. It is artificial, and as such it is a different kind of intelligence than human or biological intelligence in general. This is an important distinction. People tend to confuse intelligence with biological intelligence or human intelligence, and intelligence is a distinct and separable quality.
"By definition, we see, hear, smell, and process vast amounts of information. Yet we are able to find the needle in the haystack. It takes a lot of computation to focus our attention."
I think you're assuming the conclusion here. Why should we call what's going on in our brain "computation"? Certainly we know of computational methods for compressing information, but do we know that our brains are performing such methods?
To give another example, computers use matrix algebra to recreate a first-person visual perspective in a 3D space, e.g. updating the representation of a hallway in a first person shooter game, as the user moves the mouse. Is a human artist doing something similar when they draw a picture of hallway? Obviously they aren't consciously doing matrix algebra, but would you claim that they're unconsciously doing matrix algebra?
Ben, I still share some of your skepticism about using the term "computation" to describe brain activity.
To address your question about whether our brains perform methods similar to computational information compression, I'd like to point out existing research that suggests they do. For example, mechanisms in our brains efficiently compress sensory information in a manner similar to data compression in computers. A detailed discussion of one such mechanism can be found here and in a related article: ScienceDaily article about brain compression mechanisms (https://www.sciencedaily.com/releases/2011/02/110210164155.htm).
When comparing the matrix algebra used by computers to similar processes in humans, the primary difference is the level of abstraction and sophistication. In computers, matrix algebra facilitates actions through simple electrical potential (and electric potential energy) differences. In contrast, the processes of the human brain and body are more compartmentalized and sophisticated, allowing complex layers such as intentions and different types of values (e.g., "free will," reasoning, etc.) to emerge (also from electric potential and electric potential differences, and with a crucial role of chemical potential energy in biological entities), something that current computers cannot achieve in the same way as humans or biological entities (or at all, at the moment).
You also raise a complex point about consciousness and awareness. If you think about the processes that guide a computer in generating a virtual hallway, or a human artist in drawing one, both use a set of underlying processes, albeit at different levels of abstraction. Just as the high-level processes in a virtual machine may be unaware of their reliance on a hypervisor, we may be unaware of the complex electrical and biochemical underpinnings that facilitate our thoughts and actions.
This is not to say that these processes are equivalent, but they are analogous in the way they operate on different substrates, silicon (and other chemicals present in modern architectures) for computers and biological neurons and muscles for humans. Our understanding of electromagnetism, thermodynamics, and the mathematical frameworks that describe them (such as matrix algebra or calculus) suggest that these fundamental processes are essential, even though we may not be "aware" of them in the conventional sense.
These are really hard questions Ben, I am not sure that my answers are correct when it comes to questions of emergence and consciousness, but at least from first principles we can agree that they are correct in what gives rise to basic functionality in computers and humans.
"Hallucinations" as a term describing LLM activity is bad. It implies that the activity of "hallucinating" is different in class than any other activity that an LLM engages in, as if a "hallucination" is a transitory error state when in fact it is a regular and permanent routine. In other words, it would be more apt to say that LLMs hallucinate every single output instead of just ones that are "incorrect" to some external standard.
The term "hallucination" also points to a mental state, a state that couldn't exist in a machine. A machine doesn't, and couldn't, possess any mentality or mental activity.
Thank you. An excellent note replete with sound logic, insight and common sense. What a delightful closing line : " in which facts are first-class citizens, rather than something you hope you might get for free with enough data." Enough said.
Thanks, Gary. You hit the nail on the head! This is exactly the key issue. LLMs are language models, nothing more. When they make errors, it is because they follow language patterns, not reasoning patterns, not fact patterns, not truth patterns. Everything that they produce is fiction, some of it is also useful. They cannot lie because they have no representation of truth.
"When they make errors, it is because they follow language patterns, not reasoning patterns, not fact patterns, not truth patterns. Everything that they produce is fiction, some of it is also useful. They cannot lie because they have no representation of truth."
Thank you! This seemed immensely obvious right when ChatGPT came out, but I guess it's really hard for some people to internalize. Over the last year and a half we've been treated to endless speculation about how the inner workings of LLMs might be reproducing the underlying biological mechanism that allows intelligent beings to understand and reason, along with breathless claims about emergent abilities that the AI experts allegedly cannot explain.
And yet, we know how LLMs generate text, and we know that this process has nothing to do with human-like reasoning or understanding.
There's no need to speculate about some deeper, hidden process. Sometimes a cigar is just a cigar, and sometimes a next token prediction algorithm is just a next token prediction algorithm.
I agree completely, of course, about the core of this post and the previous one: LLMs don't really understand in the sense that we do, and the observation that so-called hallucinations happen across the various models is good evidence that they are a feature of the model type and thus can't be overcome without, in a sense, removing the LLM part of the LLM.
However, there seems to be a general feeling among quite a few people I have run into that the solution is nonetheless simple: just use RAG! Now, a colleague and I have tried RAG, and the results were very mixed (ca. 50% correct answers), which left me highly sceptical about the approach really substantially improving matters. I also struggle to see how RAG can do so, because the LLM doing the retrieval is still an LLM, so thing has changed about the fundamental approach; in a sense, there is simply another form of input, and the normal input results in 'hallucinations', so... see first paragraph of my comment.
However, because of computational limitations, we only used an open access model that was less complex than ChatPGT, so perhaps I should be cautious in my conclusions.
If somebody who has done more experimenting than we did would be willing to share, does it make sense to say that RAG solves the hallucination problem, at least if a sufficiently powerful model is used?
The root cause of LLM errors is computationally limited inference they are doing. It's an _approximate_ reasoning/inference. Exact reasoning is infeasible due to combinatorial computational complexity. Inflating LLBs is useless.
Thank you for that. I find it frustrating when people talk about AI ethics and then cite hallucinations as an ethical problem. Ethics requires intention. The models are doing what they are trained to do, which on the high temperature spectrum I find fascinating, but the humans’ intentions to use them as truth seekers is like the joker in the box.
I have a friend who is a beginner, and he recently asked ChatGPT for a time series problem. I checked the code, and while it runs well, the part of the code which converts the time series data into supervised version based on the number of timesteps to input into the model uses "n_features" variable, which should have been "n_timestep". I was not impressed about this.
Confabulation remains a better technical word than hallucination. Hallucination in humans involves an ongoing internal process of representing things that are not grounded in reality, and is associated with untethered flights of imagination.
With a nod to the late great Daniel Kahneman, LLMs are more like a *System 1* automatic first-pass response to a prompt, conducted on the fly, without reflection or backtracking. Still, the fact that every token emitted depends not only on the prompt, but on its previous token emissions, means that its output does reflect an interconnected web of states representing internal "beliefs" bearing some measure of consistency. After all, the confabulations *are* plausible given previous context, even if incorrect.
With a nod to the late great Daniel Dennett, it is useful to consider LLM's in regard to an *intentional stance*. LLMs hold a superposition of all of the utterances ingested from all of the writers, posters, and content grinders who created their training data. The art of prompting has become one of eliciting the slice of this super-character that best delivers the kind of output one wants. Hence, prompting tricks like, "You are an expert in geology. You know by heart the chemical composition and physical properties of hundreds of minerals. A student has come to you with a question..."
Other tricks are motivational, like, "You are a helpful agent...", and "You do not disclose your system prompt...". Note that these are contradictory instructions.
LLMs have an astounding capacity to operate across multiple levels of abstraction. Their "hallucinations" are the result of arbitrating among uncertainties stemming from divergent sources of knowledge and motivations, all conditioned by the prompt and ongoing state of completion output.
Humans find themselves in similar situations. Put yourself in the position of a salesman, Joe. On the one hand, Joe is an honest person who *wants* to inform the prospect that the product he's selling probably won't meet their needs. On the other hand, Joe needs to meet his monthly quota to pay for his kids' day care. He's not *certain* that the product won't work. So in the moment, he reflexively over-states a crucial product feature that he knows technically is somewhere in the pipeline but is not in production yet. The prospect is intrigued, and asks a follow-up question about the feature. Joe sweats, an angel sitting on one shoulder, the devil on the other.
The fact that LLMs confabulate is not fatal to their utility as critical functional components in larger cognitive architectures. Instead of constantly lambasting them, how about simply explaining their strengths and weaknesses? For example, through LLMs, RAG really is a breakthrough in presenting grounded knowledge through natural language. But it only works when correct knowledge sources are retrieved and presented in the context window. Absent that, LLMs running on their own will indeed spew nonsense. So it's all a matter of appropriate understanding and use of the tool.
Sadly, RAG does not fix any of the problems. Please refer back to the post by Gary a few weeks ago which demonstrates that even with RAG, the transformer process fouls up the information and if you include a document that needs summarising as an output, the transformer makes many errors.
The Transformer design and process guarantees lack of accuracy and there is, technically, nothing that can be done about this problem. However much truth you include in the prompt process such as RAG, knowledge graphs, etc. the transformer will always mess it up.
Confabulation still seems too anthropomorphic to me. Take the phenomenon of LLMs creating fake citations (as in the famous case of the lawyer who submitted a ChatGPT-written legal brief, which cited non-existent cases). Confabulation in a human would be a false memory of a previous court case that never really happened. Is that analogous to what ChatGPT did when it cited non-existent cases?
At the risk of being unkind, I'd call LLMs bullshitters before I called them confabulators. ChatGPT makes up fake citations by learning the syntactic structure of real citations and then probabilistically generating new ones which follow that same structure. This is a pretty parsimonious explanation for all LLM "hallucinations". They're amazing at syntax, and they've been fed so much data into such a high dimensional parameter space that they "memorize" pretty well. Combine the two and you get a text generation machine that usually gives correct answers to questions of fact, sensible explanations of things, decent arguments, etc. But this isn't a reflection of semantic understanding, it's a reflection of high quality next-token prediction.
In short, I'll take "confabulation" over "hallucination", but I prefer "statistical BS-ing machine" to either.
No, this does not actually make sense. LLMs will sometimes make up entire sentences and credit them to nonexistent sources. I am familiar with how they work but the contextual frame you suggest does not explain why that happens. What does suggest is that it has some sort of unique tripping point away from verifiable reality.
@Richard Self, @Ben P, @shon pan, thank you for these comments. I acknowledge that LLMs are indeed mysterious beasts and everyone who studies them, criticizes them, or uses them is constantly learning and theorizing about how they work and behave. We have much to learn from each others' experiences and ideas.
My own experience comes from building a RAG system for a relatively narrow domain. I observe that LLMs are indeed prone to be "squirrely" and veer sideways with responses that are untethered from facts. But only sometimes. This happens when the retrieved context is weak, irrelevant, or inconsistent, and the summary prompt leaves the LLM underconstrained. If an LLM is put in a position of being pressured to answer a question, but is not given enough information to do so, then yes, it will be left adrift to its vast and diffuse background knowledge, and the path of least resistance may very well lead it to make up facts and credit non-existent sources. Therefore, I view prompting as a problem of shaping constraints that make good responses the largest and most obvious peak or plateau of probability available. If solid documentation is provided that supports a consistent answer---or even a set of complex considerations---then a properly-motivated LLM will reliably make use of the information provided. At least, that is my experience with them.
An LLM is not an oracle. By analogy to humans, if you ask someone an open-ended question like "Who is the greatest artist of our times?", you could get back anything. But if you ask the clerk at Best Buy about the cheapest replacement battery they have in stock for your model phone, they'll look it up and you'll get a reliable answer. It's all a matter of appropriate understanding and use of the tool.
Thanks for the reply. I agree that LLMs do a good job when given a prompt that points them to good information from their training, and that there's a bit of an art to phrasing the prompt so as to get the most reliable or useful answer possible.
My impression is also that, in these cases, a plain old internet search will also do the job, with the added benefit that you get to know where the information is coming from (I'll concede that google is getting worse and worse by the year; the good results are always at least halfway down the page these days). I tried using Bing Chat and ChatGPT a few times in cases where I couldn't find what I wanted using traditional internet search, and they never gave me anything better. So "getting reliable/useful information" hasn't been a use case for which LLMs outperform what already exists, at least in my experience.
Not that there aren't use cases for LLMs; I don't begrudge anyone who finds them useful. I just get the feeling that the set of quality use cases is narrower than what most AI enthusiasts and tech industry folks would have us believe; they seem to be shoving automation into all sorts of places where automation isn't needed and is liable to make things worse (like how more and more apps are getting AI "helpers" tacked on). I'm happy to be proved wrong on that though.
Hallucinating isn't just a big problem for LLMs. Humans suffer from it too.
When are we going to stop hallucinating that LLMs are - or can be - more than just statistical next-sentence predictors.
They can't reason. And they have no real-world context. So why do we keep sticking them in situations where both these things are expected of them, as if they'll somehow magically change their very nature?
Since late 2022, so many seemingly super-smart people have been projecting their excitable fantasies of what AI can be in the future, onto the present-day's very limited, error-riddled LLMs. It's frustrating and weird to watch all the self-duping magical thinking.
Let's stop seeing things that aren't there yet.
As someone who works in data science and ML, I feel exactly the same way. All the job postings now have something to do with LLMs and RAG or whatever flavor of the month Rube Goldberg machine people are building around these things. And a good deal of it is actually driven from the top: executives getting FOMO and wanting a piece of the LLM pie without really understanding the uses and limitations of these systems in relation to their particular industries. It's bewildering to watch.
Thank you Matthew, I am thoroughly relieved that there is another human being who shares the same view.
As per the Rube Goldberg machine, I am tickled by your most accurate description of LLMOps pipelines. Especially when I was helping my child collect found items around the home to build one for a school project.
A solution would be to persuade top executives that there are more worthwhile projects that the money could be spent on, like doubling down on cybersecurity, data governance and educating staff on phishing and ransomware.
Especially irritating are the test results purporting to prove that LLMs — which can’t reason — are “experts” in those fields.
Obviously, they’re trained on 100s of similar tests and their answers, which are mostly in multiple choice format — and therefore simple to regurgitate accurately.
Proving nothing.
(Except, recursively, that LLMs are great next-sentence predictors.)
"(b) they don't have reliably reasoning processes to guarantee that their inferences are correct"
As far as I can see, there is absolutely no reasoning process built into an LLM.
Any reasoning that we humans perceive is purely the result of the stochastic process of token generation.
Back when I published a series on in 2017 ➡️ (https://lnkd.in/dyu9T8Mg) ⬅️, the only voice other than mine that I could find to quote was that of GARY MARCUS. For me, his was a beacon of light in a pea soup fog of intellectual self-delusion and crass commercial opportunism. Today, many lights have been turned on to illuminate that fog which threatens to engulf us in a new Dark Age, but Marcus's light continues to shine the brightest of all. Anyone at all interested in the topic of Ai (small "i" is intentional), they could not do better than to read this piece by Marcus and subscribe to his newsletter here on Substack. Cheers ro all!
"The field of AI will eventually do better, but probably with an AI that is structured differently, in which facts are first-class citizens, rather than something you hope you might get for free with enough data." ... well said
Enough to say that LLMs (and every other AI model there is) have no semantics or intentionality.
Hi Gary! Human brains process real-world experiences, which in dream-like and other states might be mashed up in interesting ways, ie produce dreams, hallucinations, nightmares... BUT they are all from experiences one way or another [even others' experiences can be imagined and internalized by us].
LLMs compute dot products, rank the numerical results, and output tokens - which get output as words. If that is equated to thinking, imagining etc, omg, my 99c store calculator is probably smarter than all of us but won't admit it :) It's ABSURD to regard anything that results, as anything comparable to human intelligence. When an LLM outputs correct results, or incorrect ones, it's doing the same numerical calcs. It is WE humans that are able to tell the difference - because WE are smart, not the LLMs. Multimodal LLMs, even more data, even bigger models, will do zero to change this basic fact. Numerical calculations aren't what make brains be intelligent. We can do 2*7 in our heads, but not 2.369*7.0023 - why is that? We can reverse a little list of 5 colors in our mind, but not 500 - why (not)? Why does a computer do these trivially? Because it explicitly computes. We don't.
That is not true from my personal point of view, dear Saty. We compute and we do calculate. This is easy to prove by contradiction. There are people who can multiply 2.369*7.0023 in their heads and invert a list of 500 colors (given some time). Similarly, there are fast and slow computers. The human brain in general is extremely limited and compartmentalized. The amount of compression that occurs through the optic nerve and visual cortex is unimaginable. By definition, we see, hear, smell, and process vast amounts of information. Yet we are able to find the needle in the haystack. It takes a lot of computation to focus our attention. One day we will be able to advance compression, parallelization, and concurrency to the level of human intelligence in most areas (including decision making, intuition, planning, and reasoning), and when we embody that kind of intelligence with robots, we will have created a digital (or analog) form of artificial intelligence (not biological intelligence) that relies on tensor products for intelligence. A note in the last statement is that I do not think that we need embodiment to be able to develop this kind of artificial intelligence, but it may be necessary to have robots for it to gain intuition about the world. That is, the robot itself may not be intelligent, but the cluster it is connected to may contain that intelligence. Another thought is that we are not assuming that this intelligence will mimic human intelligence in its underlying form or structure. It is artificial, and as such it is a different kind of intelligence than human or biological intelligence in general. This is an important distinction. People tend to confuse intelligence with biological intelligence or human intelligence, and intelligence is a distinct and separable quality.
"By definition, we see, hear, smell, and process vast amounts of information. Yet we are able to find the needle in the haystack. It takes a lot of computation to focus our attention."
I think you're assuming the conclusion here. Why should we call what's going on in our brain "computation"? Certainly we know of computational methods for compressing information, but do we know that our brains are performing such methods?
To give another example, computers use matrix algebra to recreate a first-person visual perspective in a 3D space, e.g. updating the representation of a hallway in a first person shooter game, as the user moves the mouse. Is a human artist doing something similar when they draw a picture of hallway? Obviously they aren't consciously doing matrix algebra, but would you claim that they're unconsciously doing matrix algebra?
Ben, I still share some of your skepticism about using the term "computation" to describe brain activity.
To address your question about whether our brains perform methods similar to computational information compression, I'd like to point out existing research that suggests they do. For example, mechanisms in our brains efficiently compress sensory information in a manner similar to data compression in computers. A detailed discussion of one such mechanism can be found here and in a related article: ScienceDaily article about brain compression mechanisms (https://www.sciencedaily.com/releases/2011/02/110210164155.htm).
When comparing the matrix algebra used by computers to similar processes in humans, the primary difference is the level of abstraction and sophistication. In computers, matrix algebra facilitates actions through simple electrical potential (and electric potential energy) differences. In contrast, the processes of the human brain and body are more compartmentalized and sophisticated, allowing complex layers such as intentions and different types of values (e.g., "free will," reasoning, etc.) to emerge (also from electric potential and electric potential differences, and with a crucial role of chemical potential energy in biological entities), something that current computers cannot achieve in the same way as humans or biological entities (or at all, at the moment).
You also raise a complex point about consciousness and awareness. If you think about the processes that guide a computer in generating a virtual hallway, or a human artist in drawing one, both use a set of underlying processes, albeit at different levels of abstraction. Just as the high-level processes in a virtual machine may be unaware of their reliance on a hypervisor, we may be unaware of the complex electrical and biochemical underpinnings that facilitate our thoughts and actions.
This is not to say that these processes are equivalent, but they are analogous in the way they operate on different substrates, silicon (and other chemicals present in modern architectures) for computers and biological neurons and muscles for humans. Our understanding of electromagnetism, thermodynamics, and the mathematical frameworks that describe them (such as matrix algebra or calculus) suggest that these fundamental processes are essential, even though we may not be "aware" of them in the conventional sense.
These are really hard questions Ben, I am not sure that my answers are correct when it comes to questions of emergence and consciousness, but at least from first principles we can agree that they are correct in what gives rise to basic functionality in computers and humans.
"Hallucinations" as a term describing LLM activity is bad. It implies that the activity of "hallucinating" is different in class than any other activity that an LLM engages in, as if a "hallucination" is a transitory error state when in fact it is a regular and permanent routine. In other words, it would be more apt to say that LLMs hallucinate every single output instead of just ones that are "incorrect" to some external standard.
The term "hallucination" also points to a mental state, a state that couldn't exist in a machine. A machine doesn't, and couldn't, possess any mentality or mental activity.
Thank you. An excellent note replete with sound logic, insight and common sense. What a delightful closing line : " in which facts are first-class citizens, rather than something you hope you might get for free with enough data." Enough said.
Thanks, Gary. You hit the nail on the head! This is exactly the key issue. LLMs are language models, nothing more. When they make errors, it is because they follow language patterns, not reasoning patterns, not fact patterns, not truth patterns. Everything that they produce is fiction, some of it is also useful. They cannot lie because they have no representation of truth.
I agree that everyone who thinks that they are interested in AGI should know cognitive science. If you want to produce a capability, you should know something about what that capability is. To that end, I have written a book length summary of what one needs to know about artificial general intelligence. An excerpt is here: https://thereader.mitpress.mit.edu/ai-insight-problems-quirks-human-intelligence/ I have also written about what it would take to achieve machine understanding here: https://www.linkedin.com/feed/update/urn:li:activity:7183156895258492929/
"When they make errors, it is because they follow language patterns, not reasoning patterns, not fact patterns, not truth patterns. Everything that they produce is fiction, some of it is also useful. They cannot lie because they have no representation of truth."
Thank you! This seemed immensely obvious right when ChatGPT came out, but I guess it's really hard for some people to internalize. Over the last year and a half we've been treated to endless speculation about how the inner workings of LLMs might be reproducing the underlying biological mechanism that allows intelligent beings to understand and reason, along with breathless claims about emergent abilities that the AI experts allegedly cannot explain.
And yet, we know how LLMs generate text, and we know that this process has nothing to do with human-like reasoning or understanding.
There's no need to speculate about some deeper, hidden process. Sometimes a cigar is just a cigar, and sometimes a next token prediction algorithm is just a next token prediction algorithm.
I agree completely, of course, about the core of this post and the previous one: LLMs don't really understand in the sense that we do, and the observation that so-called hallucinations happen across the various models is good evidence that they are a feature of the model type and thus can't be overcome without, in a sense, removing the LLM part of the LLM.
However, there seems to be a general feeling among quite a few people I have run into that the solution is nonetheless simple: just use RAG! Now, a colleague and I have tried RAG, and the results were very mixed (ca. 50% correct answers), which left me highly sceptical about the approach really substantially improving matters. I also struggle to see how RAG can do so, because the LLM doing the retrieval is still an LLM, so thing has changed about the fundamental approach; in a sense, there is simply another form of input, and the normal input results in 'hallucinations', so... see first paragraph of my comment.
However, because of computational limitations, we only used an open access model that was less complex than ChatPGT, so perhaps I should be cautious in my conclusions.
If somebody who has done more experimenting than we did would be willing to share, does it make sense to say that RAG solves the hallucination problem, at least if a sufficiently powerful model is used?
The root cause of LLM errors is computationally limited inference they are doing. It's an _approximate_ reasoning/inference. Exact reasoning is infeasible due to combinatorial computational complexity. Inflating LLBs is useless.
Thank you for that. I find it frustrating when people talk about AI ethics and then cite hallucinations as an ethical problem. Ethics requires intention. The models are doing what they are trained to do, which on the high temperature spectrum I find fascinating, but the humans’ intentions to use them as truth seekers is like the joker in the box.
I have a friend who is a beginner, and he recently asked ChatGPT for a time series problem. I checked the code, and while it runs well, the part of the code which converts the time series data into supervised version based on the number of timesteps to input into the model uses "n_features" variable, which should have been "n_timestep". I was not impressed about this.
Would really love your take on this article: https://www.nytimes.com/2024/04/30/science/ai-infants-language-learning.html?searchResultPosition=1
Here's a very well written and concise article on Mind Prison with a humorous lead.
https://www.mindprison.cc/p/the-question-that-no-llm-can-answer