LLMs are actually *Large Word Sequence Models*. They excel at reproducing sentences that sound correct. Mostly because they have been trained on billions of small groups of word sequences.
However language exists to transfer meaning between humans. Calling Chatbot a LLM implies it conveys meaning. Any meaning and correctness behind these generated word sequences is purely incidental and any potential meaning is inferred solely by the reader.
Saying that, Chatbot is ground-breaking technology, it will help the non-English speaking with syntax and grammar. But it will help no-one with conveying meaning.
When the next generation looks back in 15 yrs and sees the $Ts poured into LLMs and non-symbolic algorithms they will be stunned at how short-sighted and misguided we currently are.
Well said, again. The level of BS we will have to endure because of the fact that these 'word order prediction systems' can produce 'correct nonsense' is really mind boggling and not many are aware of the scale of the problem. So, good that it is pointed out.
With respect to: what should we do about it: I would humbly suggest people to listen to the last 7 minutes of my 2021 talk: https://www.youtube.com/watch?v=9_Rk-DZCVKE&t=1829s (links to last 7 minutes) it discusses the fundamental vulnerability of human intelligence/convictions and the protection of truthfulness as a key challenge of the IT revolution.
Also in that segment: one thing we might do at minimum is establish a sort of 'Hippocratic Oath for IT'. And criminalising systems pretending to be human.
There is more and those were first thoughts (though before 2000 I've already argued that internet anonymity when 'publishing' will probably not survive the fact that it enables damage to society too much)
Final quote from that 7 minute segment at the end of the talk:
"It is particularly ironic is [sic] that a technology — IT — that is based on working with the values 'true' and 'false' (logic) has consequences that undermine proper working of the concepts of 'true' and 'false' in the real world."
Harry G. Frankfurt famously defined "bullshit" as speech intended to persuade without regard to the truth. This is different from lies, which are deliberate departures from truth, and thus require the liar to refer to truth in some way, as part of their action. The liar knows the truth and conceals it. The bullshitter doesn't make any attempt to ascertain true or false.
By this definition, any and all products of large language models are bullshit. Regardless of their use or the intentions of their users.
Wondering whether Frankfurt's definition of the B.S. problem might be a better way to define the trouble with LLMs -- rather than truth v. lies. The latter opens you to replies along the lines of "but it's often right!" or "ethical people will use it ethically!"
Dec 10, 2022·edited Dec 10, 2022Liked by Gary Marcus
Great piece Gary, thanks for sharing. The challenge of finding ways to integrate large language models with "reasoning" is a significant barrier that requires entirely new concepts and approaches. This challenge is made much more complex since there is little understanding or agreement on what reasoning is. And there is little meaningful scalable success in defining how computers and software could reason, or at least safely emulate reasoning. But the need for good reasoning, by both humans and computers systems, and then how they could collaborate using reasoning, is essential and the urgency for this has just been profoundly escalated.
OMG Gary, I love your 4-point proposal! Indeed, good old search (eg for 'kitten' or 'Picasso') is in danger of getting polluted if the search engine indexes newer generative AI creations. Any 'news' item [eg about the discovery of a new species of butterfly, complete with pretty pics] is (should be) automatically sus - even if it is "actually" true! That's a sad world to live in.
Every AI generator to date, including the ones that create pretty artwork and output chat, is computing and outputting symbols, that's all it does. Those symbols do not, CANNOT be grounded in reality - because the generators have no way to relate their input symbols to anything they 'know', which in turn is because they have no first-hand experience.
We have ELIZA++ that outputs grammatically correct text in much longer sequences than Weizenbaum's original program, but that's about it.
****
Me: How can someone without a body interact with you?
ChatGPT: As a machine learning model, I exist solely as a collection of algorithms and data, and do not have a physical form or presence. Because of this, I cannot be directly interacted with in the same way that a physical object or a human with a physical body could be. Instead, I am designed to process and analyze text-based inputs in order to generate responses and provide information on a wide range of topics. Someone without a body could interact with me by providing text-based input, such as a written or spoken question, and I would use the algorithms and data that I have been trained on to generate a response. This response could then be read or heard by the person without a body, allowing for a form of interaction even without a physical body.
****
Like you point out, this is dangerous, more than it is amusing.
We simply need to evolve the internet culture to only trust IPFS content that is signed with a public key. These public keys are linked to DIDs in a decentralized reputation graph. All other content will have to be assumed autogenerated by bots.
Thanks Gary for making helpful points that these Generative-Pretrained-Transformer AI systems, like ChatGPT, are simultaneously very fun to use and yet (1) make many mistakes, so user beware, (2) can be weaponized by bad actors, and (3) are inexpensive to use by bad actors and other users alike. My further opinion here https://service-science.info/archives/6309
While it is completely true that ChatGPT generated text looks more human like and plausible, my question is "What prevents someone from intentionally or unintentionally promoting misinformation on the web?" Isn't this already a problem of the internet? Even before ChatGPT, I have read a whole lot of articles and videos which I found were not accurate. Internet content can only be trusted so much. The trust factor has just decreased to an even lower level.
The problem is not of content generated v/s content written. The problem is of the gullibility of people when they read the content. It is like believing ads on TV when some movie person promotes it, or the way the crypto market moved from highs to lows based on actions and tweets of famous personalities. People learnt not to trust anything & everything and things started stabilising.
My take, this is just the initial phase of chatGPT. I am sure people will start disregarding and start developing the required fences around what they read and trust. Who knows, the addiction to social media and internet may start subsiding which just maybe a blessing in disguise for human civilisation as a whole.
I think we should let it run free. I’d love for every person on earth to have access to a full and uncensored version of ChatGPT that could scour the net and other types of data and conduct robust research.
Humans can figure out misinformation vs what’s real. We all sort of know what is fake. Don’t underestimate humans.
This could be a very powerful tool for positive change if we allow everyone equal access to it.
Just like the Internet itself, of course some will abuse this power, but the overwhelming majority of people are “good” people that don’t want to harm others. We will mostly use this for good and for innovation and to build wealth.
It's worth remembering what happened to machine translation in the mid-1960. MT, as it was known, was going great guns in the 1950s. Alas, many researchers were also making promises they were unable to fulfill. High-quality MT is just around the corner, things like that. By the early 1960s the federal government, which had been footing the bill, was wondering when the promised results were going to materialize. They wanted those Russian technication documents translated into English, now!
So in 1964 they appointed a blue-ribbon panel to investigate, the Automatic Language Processing Advisory Committee, known as ALPAC. (FWIW, my teacher, David Hays, was on that committee.) The committee, in effect, came up with two recommendations: 1) There is no near-term prospect of high-quality machine translation so scratch that. 2) But we now have theoretical concepts we didn't have when we'd first started, so now's the time to fund basic research.
If you don't already know, you can guess what happened. The government took the first recommendation, but ignored the second. Funding dried up. Though MT was at the time a separate enterprise from AI, and still is more or less, you can think of that as the first AI Winter. (BTW, and that's how the field came to be known as "computational linguistics." It was rebranded.)
Machine learning these days seems mostly funded by private enterprise. That's certainly true for the really large projects, the ones of "foundational" scope. Still, if there's a nasty public backlash for the reasons you suggest, Gary, I can imagine the results will be much the same as they were in the 1964s. Note that back in the 1960s very few people had even seen a computer, much less held one in their hand as an everyday tool. MT was unknown to the public beyond a quickly forgotten news story.
I find it hard to imagine that those companies would be willing to fund the kind of research you favor, as do I, in the face of withering disdain from a populace that is outraged at what they've done. They're not funding it now, why should they do any different in the face of an angry public, a public that could care less about the (esoteric) difference between pure DL and hybrid systems? Nor would the Federal government be in much of a mood to provide funding.
Is there an opportunity here? What if a company established a "Validation Service"?
An author would get a "validation link" for something they wrote. If the reader found the information valuable enough, but wanted to confirm the source and check the facts - they would click on the validation link. There would be a charge, depending on the size of the article.
If enough people paid for validation - perhaps the charge would be zero. And then the article would get a validation "seal".
Perhaps the firm that the author wrote for would pay for a validation seal up front.
Articles without a validation link or seal could be ignored.
Is it possible to create a digital watermark that could be used to identify AI-authored material? I'd like to see a "caveat" emoji as well that warns readers against questionable stuff whether computer- or human-generated.
LLMs are actually *Large Word Sequence Models*. They excel at reproducing sentences that sound correct. Mostly because they have been trained on billions of small groups of word sequences.
However language exists to transfer meaning between humans. Calling Chatbot a LLM implies it conveys meaning. Any meaning and correctness behind these generated word sequences is purely incidental and any potential meaning is inferred solely by the reader.
Saying that, Chatbot is ground-breaking technology, it will help the non-English speaking with syntax and grammar. But it will help no-one with conveying meaning.
When the next generation looks back in 15 yrs and sees the $Ts poured into LLMs and non-symbolic algorithms they will be stunned at how short-sighted and misguided we currently are.
Well said, again. The level of BS we will have to endure because of the fact that these 'word order prediction systems' can produce 'correct nonsense' is really mind boggling and not many are aware of the scale of the problem. So, good that it is pointed out.
With respect to: what should we do about it: I would humbly suggest people to listen to the last 7 minutes of my 2021 talk: https://www.youtube.com/watch?v=9_Rk-DZCVKE&t=1829s (links to last 7 minutes) it discusses the fundamental vulnerability of human intelligence/convictions and the protection of truthfulness as a key challenge of the IT revolution.
Also in that segment: one thing we might do at minimum is establish a sort of 'Hippocratic Oath for IT'. And criminalising systems pretending to be human.
There is more and those were first thoughts (though before 2000 I've already argued that internet anonymity when 'publishing' will probably not survive the fact that it enables damage to society too much)
Final quote from that 7 minute segment at the end of the talk:
"It is particularly ironic is [sic] that a technology — IT — that is based on working with the values 'true' and 'false' (logic) has consequences that undermine proper working of the concepts of 'true' and 'false' in the real world."
Harry G. Frankfurt famously defined "bullshit" as speech intended to persuade without regard to the truth. This is different from lies, which are deliberate departures from truth, and thus require the liar to refer to truth in some way, as part of their action. The liar knows the truth and conceals it. The bullshitter doesn't make any attempt to ascertain true or false.
By this definition, any and all products of large language models are bullshit. Regardless of their use or the intentions of their users.
Wondering whether Frankfurt's definition of the B.S. problem might be a better way to define the trouble with LLMs -- rather than truth v. lies. The latter opens you to replies along the lines of "but it's often right!" or "ethical people will use it ethically!"
Great piece Gary, thanks for sharing. The challenge of finding ways to integrate large language models with "reasoning" is a significant barrier that requires entirely new concepts and approaches. This challenge is made much more complex since there is little understanding or agreement on what reasoning is. And there is little meaningful scalable success in defining how computers and software could reason, or at least safely emulate reasoning. But the need for good reasoning, by both humans and computers systems, and then how they could collaborate using reasoning, is essential and the urgency for this has just been profoundly escalated.
https://twitter.com/GnosisEngineer1/status/1601609189973188614?s=20&t=xtYLAKqN0w3Jxj_9PEtCiw
OMG Gary, I love your 4-point proposal! Indeed, good old search (eg for 'kitten' or 'Picasso') is in danger of getting polluted if the search engine indexes newer generative AI creations. Any 'news' item [eg about the discovery of a new species of butterfly, complete with pretty pics] is (should be) automatically sus - even if it is "actually" true! That's a sad world to live in.
Every AI generator to date, including the ones that create pretty artwork and output chat, is computing and outputting symbols, that's all it does. Those symbols do not, CANNOT be grounded in reality - because the generators have no way to relate their input symbols to anything they 'know', which in turn is because they have no first-hand experience.
We have ELIZA++ that outputs grammatically correct text in much longer sequences than Weizenbaum's original program, but that's about it.
****
Me: How can someone without a body interact with you?
ChatGPT: As a machine learning model, I exist solely as a collection of algorithms and data, and do not have a physical form or presence. Because of this, I cannot be directly interacted with in the same way that a physical object or a human with a physical body could be. Instead, I am designed to process and analyze text-based inputs in order to generate responses and provide information on a wide range of topics. Someone without a body could interact with me by providing text-based input, such as a written or spoken question, and I would use the algorithms and data that I have been trained on to generate a response. This response could then be read or heard by the person without a body, allowing for a form of interaction even without a physical body.
****
Like you point out, this is dangerous, more than it is amusing.
We simply need to evolve the internet culture to only trust IPFS content that is signed with a public key. These public keys are linked to DIDs in a decentralized reputation graph. All other content will have to be assumed autogenerated by bots.
Thanks Gary for making helpful points that these Generative-Pretrained-Transformer AI systems, like ChatGPT, are simultaneously very fun to use and yet (1) make many mistakes, so user beware, (2) can be weaponized by bad actors, and (3) are inexpensive to use by bad actors and other users alike. My further opinion here https://service-science.info/archives/6309
While it is completely true that ChatGPT generated text looks more human like and plausible, my question is "What prevents someone from intentionally or unintentionally promoting misinformation on the web?" Isn't this already a problem of the internet? Even before ChatGPT, I have read a whole lot of articles and videos which I found were not accurate. Internet content can only be trusted so much. The trust factor has just decreased to an even lower level.
The problem is not of content generated v/s content written. The problem is of the gullibility of people when they read the content. It is like believing ads on TV when some movie person promotes it, or the way the crypto market moved from highs to lows based on actions and tweets of famous personalities. People learnt not to trust anything & everything and things started stabilising.
My take, this is just the initial phase of chatGPT. I am sure people will start disregarding and start developing the required fences around what they read and trust. Who knows, the addiction to social media and internet may start subsiding which just maybe a blessing in disguise for human civilisation as a whole.
This thing is dangerous. I propose a moratorium- except for carefully controlled research - until strong regulation is in place
I think we should let it run free. I’d love for every person on earth to have access to a full and uncensored version of ChatGPT that could scour the net and other types of data and conduct robust research.
Humans can figure out misinformation vs what’s real. We all sort of know what is fake. Don’t underestimate humans.
This could be a very powerful tool for positive change if we allow everyone equal access to it.
Just like the Internet itself, of course some will abuse this power, but the overwhelming majority of people are “good” people that don’t want to harm others. We will mostly use this for good and for innovation and to build wealth.
That is my belief.
thanks; fixed; https://twitter.com/dileeplearning/status/1599240840454758401?s=20&t=z1zDwZh4sBDmomiWWuHQzw
oops ficing; https://twitter.com/dileeplearning/status/1599240840454758401?s=20&t=z1zDwZh4sBDmomiWWuHQzw
It's worth remembering what happened to machine translation in the mid-1960. MT, as it was known, was going great guns in the 1950s. Alas, many researchers were also making promises they were unable to fulfill. High-quality MT is just around the corner, things like that. By the early 1960s the federal government, which had been footing the bill, was wondering when the promised results were going to materialize. They wanted those Russian technication documents translated into English, now!
So in 1964 they appointed a blue-ribbon panel to investigate, the Automatic Language Processing Advisory Committee, known as ALPAC. (FWIW, my teacher, David Hays, was on that committee.) The committee, in effect, came up with two recommendations: 1) There is no near-term prospect of high-quality machine translation so scratch that. 2) But we now have theoretical concepts we didn't have when we'd first started, so now's the time to fund basic research.
If you don't already know, you can guess what happened. The government took the first recommendation, but ignored the second. Funding dried up. Though MT was at the time a separate enterprise from AI, and still is more or less, you can think of that as the first AI Winter. (BTW, and that's how the field came to be known as "computational linguistics." It was rebranded.)
Machine learning these days seems mostly funded by private enterprise. That's certainly true for the really large projects, the ones of "foundational" scope. Still, if there's a nasty public backlash for the reasons you suggest, Gary, I can imagine the results will be much the same as they were in the 1964s. Note that back in the 1960s very few people had even seen a computer, much less held one in their hand as an everyday tool. MT was unknown to the public beyond a quickly forgotten news story.
I find it hard to imagine that those companies would be willing to fund the kind of research you favor, as do I, in the face of withering disdain from a populace that is outraged at what they've done. They're not funding it now, why should they do any different in the face of an angry public, a public that could care less about the (esoteric) difference between pure DL and hybrid systems? Nor would the Federal government be in much of a mood to provide funding.
Is there an opportunity here? What if a company established a "Validation Service"?
An author would get a "validation link" for something they wrote. If the reader found the information valuable enough, but wanted to confirm the source and check the facts - they would click on the validation link. There would be a charge, depending on the size of the article.
If enough people paid for validation - perhaps the charge would be zero. And then the article would get a validation "seal".
Perhaps the firm that the author wrote for would pay for a validation seal up front.
Articles without a validation link or seal could be ignored.
Is it possible to create a digital watermark that could be used to identify AI-authored material? I'd like to see a "caveat" emoji as well that warns readers against questionable stuff whether computer- or human-generated.