To communicate interlocutors need to share some context. To lie, the liar needs to have a private context unknown to the other party. If lie is about the shared context and can be easily called, it is not lie, it is stupidity.
When someone says GPT is lying, they are granting it a distinctly human trait. A trait it clearly does not have.
I'd suggest that both Santos and GPT had faulty training data. Maybe Santos believes that American business is based on lies, so he feels no guilt when he lies. I won't deal with companies that lie to me, but that is another subject.
GPT seems capable of producing good code, perhaps it's because there is less trash code on the Internet.
For other subjects the amount of careless data, incompetent ramblings and outright lies is much greater. GPT does not have the ability to discern the validity of that data, and that is the source of its less than accurate results.
Lying is intentional. Similarly immorality is intentional. It is impossible to be immoral by accident. Machines are not immoral; even animals are not immoral. They do what they do inexorably. Dogs bite children, lions kill babies, neither is morally culpable. Dogs are dogs; lions are lions; machines are machines. That is the foundation of morality: conscious intent. To lie, which in most instances is immoral, is to know you are intentionally deceiving another for an unjustifiable reason. Accidental falsehoods, however, are without intent; one is not morally culpable for lying if the falsehood is a matter of stupidity or ignorance or even a lack of due diligence. Santos is a liar; he intended to lie. ChatGPT is amoral; it intends nothing. Chat has no intentions at all, because it is entirely without consciousness. But we who are foolish enough not to do our due diligence are culpable, not for lying but for, as we will see as law suits arise, criminal negligence. We all know Chat is not conscious. It cannot lie. We know damn well it does not care about anything including the truth. Yet when we put lives in the hands of uncaring machines and people die, we who were negligent will pay the price. Santos will go to jail for being Santos; ChatGPT nor the creators of ChatGPT will go to jail, but the fool who deludes himself and uses ChatGPT recklessly may well serve some jail time for criminal negligence. We do not punish the sword but the one holding it. ChatGPT is a chainsaw that is hard to control, but damn does it cut.
Well said. Excellent piece. Like the broken thermometer that indicates the right temperature by accident every once-in-a-while. No intention and no useful information except by chance. But by being wrong - lying - and learning from being wrong, perhaps deep neural networks will acquire intentionality at some point. Safe to say the creators of AGI must be working toward that goal. The scary part is they may be hopeful for mistakes - and lying - as part of the evolution of intelligence.
Now we are talking turkey, and right on time! Intentionality entails caring. We conscious creatures are in a constant state of caring intentionality. We cannot look at anything without in some way tying it back to our own biological reality. We like it, we do not like it, we want it, we are disgusted by it. We are actually incapable of ignoring anything. For a consciousness to ignore something is to care about it by intentionally not caring about it. Seems a paradox, but try and ignore your ex-spouse at the table across the room and not notice that you are ignoring her/him. That is paying attention to her/him, in the mode of ignoring her/him! But so far, AI does not care about anything at all. Simply because it has no desires to frustrate. It really could ignore it's ex-spouse!! It has nothing to lose, no regrets, not a care in the world. Now we could run a pretend-to-care algorithm I suppose, but that would just be a simulation of caring. The problem, I think, is AI as of yet is in the perpetual present. It is always now for ChatGPT and Bard and pals. It cannot project a future. It can very very well *predict* the future, talk to actuarial people! But AI cannot desire anything in the future. It is absolutely locked in the present. It does not even grasp its past in terms of the future. In other words, AI does not have any time consciousness whatsoever. We on the other hand are driven mad by the future in light of the past, neither of which exist in the present. Everything we do is for the future that does not yet exist and never does except as a projection based on past experience and hope mixed with dread!
Recently heard of the concept of 'data voids'. Does this apply to LLM models too? Any answer is better than no answer it seems. A bit like Trump - who hates to admit ignorance, and makes stuff up, I suspect.
The issue of lying seems to provide an interesting test case for the governance of AI. I published a case study (https://www.linkedin.com/pulse/chatgpt-when-do-hallucinations-turn-deceit-mike-baxter/) a few weeks back describing my request to ChatGPT4 for quotations on a specific topic (business strategy). Having been given several such quotations, I realised one was not actually a quotation. I challenged ChatGPT4 on this and got this response: "'I apologize for the confusion earlier, but upon further research, it seems that the line you mentioned doesn't appear to be a direct quotation from the article. I must have paraphrased their main idea rather than quoted them directly". Then, later in our interaction, it offered a "relevant direct quote from the article", which also turned out to be a hallucination. Reflecting on this issue, it dawned on me that, whilst a LLM has a syntactic comprehension of a quotation (words enclosed in quotes and attributed to a source), it cannot have a semantic understanding of a quotation (a verbatim extract from a cited source, reproduced unchanged or modestly changed and recoded as such using conventional notations). This suggests that quotations are a known case of 'very high probability hallucinations' with a clear signal - the use of the word stem 'quot*' in the prompt. This also has a clear solution - give the warning up-front that ChatGPT will eventually provide after it has been challenged "For exact quotations, always refer to the original article". Clearly this won't stop the problem of hallucinations but it would show that steps to prevent users from being misled are being introduced into AI governance practices.
Great article in LinkedIn Mike. Chat will also review an article it has not seen if you only use the title and ask it to review it. My students asked it to compare an article in Wired Magazine with a movie. It pretended to do it very very effectively since the title of the article was a giveaway of the content, and it pulled stuff from everywhere but the article itself. It did know the movie.
When you converse with a human, you take it as a given that the other party has a somewhat static persona. But when you converse with an LLM, it uses the things you say to update, in mid-conversation, the persona it is presenting to you.
This unsettling aspect of LLMs always struck me as being similar to how a con-person might interact with a mark, altering their persona or their backstory 'on the fly' to maximise the likelihood of the con succeeding.
Do you think "backpropagation" is itself the root problem here? Perhaps the loop re-propagates some sort of invisible flaw. As it washes out errors some errors may simply be impervious for some reason. Backpropagation was the revolutionary moment in AI, so it would make sense that some flaw there could be leading to the hallucinations. I do not have the mathematical understanding to know how it works but perhaps someone here could simply say no that is nonsense or maybe. (please be patient with a dolt)
There is an entirely different set of problems arising from the way in which chains of shared associations can lead to unreliable inferences about the world. For example, a colleague of mine is responsible for a (relatively obscure) theoretical hierarchy in the field of computational linguistics. If you prompt ChatGPT-4 about it, we found that for a few sentences the text produced was a lucid and accurate summary, but it then drifted into a discussion about a different hierarchy (Chomsky's hierarchy) - a different theory in the same sub-field, essentially overlapping and intermingling in a particular corner of the model's parameter space. This form of hallucination seems to be very common and fiendishly hard to spot because the hallucinations look so credible. This is very much a feature not a bug as these systems are "natural language inference" calculators; they perform their calculations through a process of inference from associative similarity.
Thank you so much for this very helpful post. I am a philosophy professor and have taught a class Robot Ethics at first year level for about 10 years. I now require my students to use an LLM on their papers. They must learn to dance with a robot. Papers have 3 parts: 1. Student creates an effective prompt. 2. Chatbot responds. 3. Student responds to bot's response. It is very easy to distinguish student writing from robot writing. One of my students recently discovered he could not get ChatGPT to doubt itself effectively. In his words: If it cannot doubt everything like Descartes does in his Cogito ergo sum argument then it is not conscious. I does NOT think so it is not.
My understanding is that the reinforcement learning played havoc with the GPT's text generation around these philosophical issues. A neat trick I read about is to ask ChatGPT to write poetry on a subject rather than respond directly; the text you get is apparently more expressive and less canned.
While I don't think hallucination is attributable to the low level backpropagation mechanism, you are correct that it is an embedded aspect of the architecture; it's a feature not a bug. Hallucinations also arise in LLMs for several different reasons which will make it very difficult to evolve the design in one particular direction to eliminate them from the core model. One set of problems emerge from the variation of density across parameter space. As a brief summary, through learning the associative relationships between tokens in a language, an LLM embeds a corpus of 'knowledge' about the world that language (generally) describes. One might call this 'common sense knowledge'. The resulting representation suffers from the "Goldilocks' Three Bears Problem". In some corners of the parameter space, there isn't any useful training data; for example, GPT-4 cannot translate the unsolved 1st and 3rd Beale ciphers for you. In other parts, the training data is sparse but sufficient to answer the question directly; GPT-4 knows that 'joy ctg aww?' means 'how are you?' because this +2 Caesar cipher has been used and decoded a few times in articles about ciphers; the system can recover reliable training data correctly. In other areas, the training data is so dense and, often, contradictory that incorrect responses might be chosen either by random or (more likely) because the 'common sense' predominant answer that emerges from the mess of data is wrong. For example, if you ask ChatGPT-4 which has more rainfall, London or Paris, it will usually tell you (incorrectly) that London has more rainfall than Paris. With introspection we can guess why this might be the 'common sense' answer; after all, the stereotypical image of an "English gentleman" used to include an umbrella on his arm whereas a parasol would come more readily to mind for a Parisian. (This is of course not the reasoning process of an LLM, I'm just trying to give an intuition why London might be much more closely associated with rainfall than Paris in the training data). So it is only in the parts of parameter space have 'just the right amount' of reliable training data that correct traces of information about the world can be recovered.
To communicate interlocutors need to share some context. To lie, the liar needs to have a private context unknown to the other party. If lie is about the shared context and can be easily called, it is not lie, it is stupidity.
When someone says GPT is lying, they are granting it a distinctly human trait. A trait it clearly does not have.
I'd suggest that both Santos and GPT had faulty training data. Maybe Santos believes that American business is based on lies, so he feels no guilt when he lies. I won't deal with companies that lie to me, but that is another subject.
GPT seems capable of producing good code, perhaps it's because there is less trash code on the Internet.
For other subjects the amount of careless data, incompetent ramblings and outright lies is much greater. GPT does not have the ability to discern the validity of that data, and that is the source of its less than accurate results.
Lying is intentional. Similarly immorality is intentional. It is impossible to be immoral by accident. Machines are not immoral; even animals are not immoral. They do what they do inexorably. Dogs bite children, lions kill babies, neither is morally culpable. Dogs are dogs; lions are lions; machines are machines. That is the foundation of morality: conscious intent. To lie, which in most instances is immoral, is to know you are intentionally deceiving another for an unjustifiable reason. Accidental falsehoods, however, are without intent; one is not morally culpable for lying if the falsehood is a matter of stupidity or ignorance or even a lack of due diligence. Santos is a liar; he intended to lie. ChatGPT is amoral; it intends nothing. Chat has no intentions at all, because it is entirely without consciousness. But we who are foolish enough not to do our due diligence are culpable, not for lying but for, as we will see as law suits arise, criminal negligence. We all know Chat is not conscious. It cannot lie. We know damn well it does not care about anything including the truth. Yet when we put lives in the hands of uncaring machines and people die, we who were negligent will pay the price. Santos will go to jail for being Santos; ChatGPT nor the creators of ChatGPT will go to jail, but the fool who deludes himself and uses ChatGPT recklessly may well serve some jail time for criminal negligence. We do not punish the sword but the one holding it. ChatGPT is a chainsaw that is hard to control, but damn does it cut.
Well said. Excellent piece. Like the broken thermometer that indicates the right temperature by accident every once-in-a-while. No intention and no useful information except by chance. But by being wrong - lying - and learning from being wrong, perhaps deep neural networks will acquire intentionality at some point. Safe to say the creators of AGI must be working toward that goal. The scary part is they may be hopeful for mistakes - and lying - as part of the evolution of intelligence.
Now we are talking turkey, and right on time! Intentionality entails caring. We conscious creatures are in a constant state of caring intentionality. We cannot look at anything without in some way tying it back to our own biological reality. We like it, we do not like it, we want it, we are disgusted by it. We are actually incapable of ignoring anything. For a consciousness to ignore something is to care about it by intentionally not caring about it. Seems a paradox, but try and ignore your ex-spouse at the table across the room and not notice that you are ignoring her/him. That is paying attention to her/him, in the mode of ignoring her/him! But so far, AI does not care about anything at all. Simply because it has no desires to frustrate. It really could ignore it's ex-spouse!! It has nothing to lose, no regrets, not a care in the world. Now we could run a pretend-to-care algorithm I suppose, but that would just be a simulation of caring. The problem, I think, is AI as of yet is in the perpetual present. It is always now for ChatGPT and Bard and pals. It cannot project a future. It can very very well *predict* the future, talk to actuarial people! But AI cannot desire anything in the future. It is absolutely locked in the present. It does not even grasp its past in terms of the future. In other words, AI does not have any time consciousness whatsoever. We on the other hand are driven mad by the future in light of the past, neither of which exist in the present. Everything we do is for the future that does not yet exist and never does except as a projection based on past experience and hope mixed with dread!
Recently heard of the concept of 'data voids'. Does this apply to LLM models too? Any answer is better than no answer it seems. A bit like Trump - who hates to admit ignorance, and makes stuff up, I suspect.
The issue of lying seems to provide an interesting test case for the governance of AI. I published a case study (https://www.linkedin.com/pulse/chatgpt-when-do-hallucinations-turn-deceit-mike-baxter/) a few weeks back describing my request to ChatGPT4 for quotations on a specific topic (business strategy). Having been given several such quotations, I realised one was not actually a quotation. I challenged ChatGPT4 on this and got this response: "'I apologize for the confusion earlier, but upon further research, it seems that the line you mentioned doesn't appear to be a direct quotation from the article. I must have paraphrased their main idea rather than quoted them directly". Then, later in our interaction, it offered a "relevant direct quote from the article", which also turned out to be a hallucination. Reflecting on this issue, it dawned on me that, whilst a LLM has a syntactic comprehension of a quotation (words enclosed in quotes and attributed to a source), it cannot have a semantic understanding of a quotation (a verbatim extract from a cited source, reproduced unchanged or modestly changed and recoded as such using conventional notations). This suggests that quotations are a known case of 'very high probability hallucinations' with a clear signal - the use of the word stem 'quot*' in the prompt. This also has a clear solution - give the warning up-front that ChatGPT will eventually provide after it has been challenged "For exact quotations, always refer to the original article". Clearly this won't stop the problem of hallucinations but it would show that steps to prevent users from being misled are being introduced into AI governance practices.
Great article in LinkedIn Mike. Chat will also review an article it has not seen if you only use the title and ask it to review it. My students asked it to compare an article in Wired Magazine with a movie. It pretended to do it very very effectively since the title of the article was a giveaway of the content, and it pulled stuff from everywhere but the article itself. It did know the movie.
When you converse with a human, you take it as a given that the other party has a somewhat static persona. But when you converse with an LLM, it uses the things you say to update, in mid-conversation, the persona it is presenting to you.
This unsettling aspect of LLMs always struck me as being similar to how a con-person might interact with a mark, altering their persona or their backstory 'on the fly' to maximise the likelihood of the con succeeding.
Do you think "backpropagation" is itself the root problem here? Perhaps the loop re-propagates some sort of invisible flaw. As it washes out errors some errors may simply be impervious for some reason. Backpropagation was the revolutionary moment in AI, so it would make sense that some flaw there could be leading to the hallucinations. I do not have the mathematical understanding to know how it works but perhaps someone here could simply say no that is nonsense or maybe. (please be patient with a dolt)
(... contd)
There is an entirely different set of problems arising from the way in which chains of shared associations can lead to unreliable inferences about the world. For example, a colleague of mine is responsible for a (relatively obscure) theoretical hierarchy in the field of computational linguistics. If you prompt ChatGPT-4 about it, we found that for a few sentences the text produced was a lucid and accurate summary, but it then drifted into a discussion about a different hierarchy (Chomsky's hierarchy) - a different theory in the same sub-field, essentially overlapping and intermingling in a particular corner of the model's parameter space. This form of hallucination seems to be very common and fiendishly hard to spot because the hallucinations look so credible. This is very much a feature not a bug as these systems are "natural language inference" calculators; they perform their calculations through a process of inference from associative similarity.
Thank you so much for this very helpful post. I am a philosophy professor and have taught a class Robot Ethics at first year level for about 10 years. I now require my students to use an LLM on their papers. They must learn to dance with a robot. Papers have 3 parts: 1. Student creates an effective prompt. 2. Chatbot responds. 3. Student responds to bot's response. It is very easy to distinguish student writing from robot writing. One of my students recently discovered he could not get ChatGPT to doubt itself effectively. In his words: If it cannot doubt everything like Descartes does in his Cogito ergo sum argument then it is not conscious. I does NOT think so it is not.
My understanding is that the reinforcement learning played havoc with the GPT's text generation around these philosophical issues. A neat trick I read about is to ask ChatGPT to write poetry on a subject rather than respond directly; the text you get is apparently more expressive and less canned.
While I don't think hallucination is attributable to the low level backpropagation mechanism, you are correct that it is an embedded aspect of the architecture; it's a feature not a bug. Hallucinations also arise in LLMs for several different reasons which will make it very difficult to evolve the design in one particular direction to eliminate them from the core model. One set of problems emerge from the variation of density across parameter space. As a brief summary, through learning the associative relationships between tokens in a language, an LLM embeds a corpus of 'knowledge' about the world that language (generally) describes. One might call this 'common sense knowledge'. The resulting representation suffers from the "Goldilocks' Three Bears Problem". In some corners of the parameter space, there isn't any useful training data; for example, GPT-4 cannot translate the unsolved 1st and 3rd Beale ciphers for you. In other parts, the training data is sparse but sufficient to answer the question directly; GPT-4 knows that 'joy ctg aww?' means 'how are you?' because this +2 Caesar cipher has been used and decoded a few times in articles about ciphers; the system can recover reliable training data correctly. In other areas, the training data is so dense and, often, contradictory that incorrect responses might be chosen either by random or (more likely) because the 'common sense' predominant answer that emerges from the mess of data is wrong. For example, if you ask ChatGPT-4 which has more rainfall, London or Paris, it will usually tell you (incorrectly) that London has more rainfall than Paris. With introspection we can guess why this might be the 'common sense' answer; after all, the stereotypical image of an "English gentleman" used to include an umbrella on his arm whereas a parasol would come more readily to mind for a Parisian. (This is of course not the reasoning process of an LLM, I'm just trying to give an intuition why London might be much more closely associated with rainfall than Paris in the training data). So it is only in the parts of parameter space have 'just the right amount' of reliable training data that correct traces of information about the world can be recovered.