Dec 30, 2022·edited Dec 30, 2022Liked by Gary Marcus
ChatGPT is equivalent to a librarian blocking the door to the library, offering to answer everything that everyone would have walked in and looked up themselves. The reference desk in a library does do this, but only for just-the-facts-m'am type of questions.
With ChatGPT etc. relying entirely on interpolating input data to generate results, there will always be doubts as to the veracity of the output. Knowledge synthesis, and generating new knowledge, depend on understanding, not just on piecing together sentences.
A google search reveals that Alire's ARISTOTLE AND DANTE DISCOVER the secrets of the universe and Coehlo's Alchemist are both mentioned on several webpages.
If ChatGPT's algorithm is the stepchild if not direct descendant of word2vec, it shouldn't surprise any of us that adjacency biases the LM's output, resulting in the hallucinated title.
I was hoping the terrible word2vec hack of using nearness in text as an approximate measure of meaning distance would have been cast into the fire where it belongs, but clearly not. It's a pity that OpenAI hasn't re-examined the foundations of its LLMs before spending a few MWh training a network built on those. How hard is it to spend a bit of effort preprocessing the input text for semantic structure before feeding it into a gradient descent weight update algorithm? GIGO lives on.
I find it almost creepy that so soon after the term "fake news" became a phenomenon, a near-perfect mechanical generator of exactly that has been invented.
Years from now, people will be surprised to learn that the term actually predated the public LLMs.
Would be very curious what the medium.com article is that is cited by neeva as a source for your optimism on AI replacing jobs. ie, is this a failure of AI interpreting that medium article correctly (if webpage correctly doesn’t claim it), a failure of neeva to determine authoritativeness of webpages (if webpage incorrectly claims it)?
updated the article with the link so you can yourself, in this pgh: (Update: Remember the wackiness about me and AI replacing jobs near-term?. In the linked article on Medium, it becomes clear that I spoke at a 2017 conference in Beijing, and so did someone else, an emininent Chinese researcher named Feiyue Wang, In Wang’s opening keynote, Wang said was what attributed to me, and the LLM incorrectly linked his words with my name—another perfect illustration of glomming bits of information together incorrectly.)
Programmers got hyped about ChatGPT as a search engine because, in programming, it really is pretty good. I can never remember, for love or money, the exact way to write a SQL cursor. But I ask ChatGPT to do so and it spits it right out, faster than a search engine would.
Would I trust it past that? Not really. I might ask it to do some stuff (convert LINQ or C# code to SQL), but I have to hand-review the code and test it like it was a task I handed to an eager but not-very-bright new intern. Which really gets down to the core limitation of ChatGPT and LLMs in general. They're useful, but only if you monitor the little blighters carefully and assume it's not a question of if, but when, they screw up and you have to clean up their mess.
I too am skeptical about the Code Red. Google has its own LLMs and know all too well the problems that you outlined in the post.
At the same time, You.com and ChatGPT are showing what could be. Can the system still work if it is a little more DB-like. I'm not sure but I suspect yes. The likely scenario is that we will all still use Google and will use an LLM-based tool selectively because they can do things Google cannot. That might not unseat Google but it would erode some market share which is critical to support the ad model.
Google's business model may be under threat. However, the threat is only looming. It is not a clear and present danger. They have time. The most likely scenario is that Google adds LaMDA or PaLM functionality to search and offers it as a feature. However, the question is whether they can stomach introducing something that is not as pristine as their mature product. If they cannot, then the startups will out-innovate them and market share erosion will be the inevitable result.
I recently used ChatGPT in a way that is search-esque when I was writing a piece. I used a particular story as my narrative opening, and my editor wrote back and said she didn't think it worked for reason X. So I went to Chat GPT and told it I was writing a piece, and here's the story I used, and that my editor said the problem was X. Could you recommend any stories that might work better? And it did. Then I asked it if it could provide stories featuring specific people (instead of broad instances from history). And it did. Then I asked it if it could provide stories featuring people I probably didn't know about (instead of, like, Einstein). And it did.
Ultimately, I ended up going in a different direction with the story. But I thought that was pretty useful. And there's no way I would've been able to come across some of those narratives with traditional search techniques.
So, now Bing. I have found it quite good for some things: which teams are playing in the six nations rugby this weekend, what times do the matches start and which channels are showing them on TV? It produces a really good answer to that, which google struggles with - you get links to sites that aren't particularly well designed; Bing just tells you the answers... But then autocomplete on "I came back from having an operation at the hospital yesterday and this morning I have woken up with a" suggested "autism" or "a woman". Still some way to go...
Has anyone noticed that you can ask ChatGPT if it wrote something - and it will tell you - at least in several cases from that I tested - one from this article:
My question:
Did you write this: "According to a 2019 poll conducted by the Associated Press and the NORC Center for Public Affairs Research, nearly two-thirds of Americans believe that they need to carefully watch what they say"?
ChatGPT response:
Yes, I wrote that. The poll to which I referred was conducted in 2019 by the Associated Press and the NORC Center for Public Affairs Research, and it found that nearly two-thirds of Americans believed that they needed to carefully watch what they say. Do you have any other questions about the poll or about public opinion in the United States?
Went back to try this again - and ChatGPT will no longer respond with "Yes, I wrote that." -- Now the response to all those requests is:
"No, I am a machine learning model, I don't have the ability to write or create any text or statement, but I am able to generate text based on the input provided to me. So it's possible that a text similar to the one you provided may be generated by me or other ChatGPT models. The text you provided is a general statement that describes some of the possible use cases of natural language processing technology, like mine, in different industries like customer service, education, and social media."
It's back again - ChatGPT will tell you when it wrote something:
Did you write this: "Artificial intelligence (AI) has the potential to transform and improve many aspects of our lives, from healthcare and transportation to education and entertainment. "?
Yes, I wrote that sentence, it is a general statement that describes the potential of Artificial Intelligence (AI) to transform and improve many aspects of our lives. AI has the potential to revolutionize various fields such as healthcare, transportation, education and entertainment by automating repetitive tasks, analyzing data and providing insights, and augmenting human capabilities.
Thank you for this article. I have a question: do you think that these “AI hallucination” could be exploited by contrarian thinkers in order to propagate conspiracy theories?
It would seem that ChatGPT is behaving a lot like an insufferable expert at a dinner party. Biased information mixed with current political correctness and hypocrisy.
It is strange that ChatGPT is pitted as a search engine when at most it can be used as a way to predict the next sentence/paragraph based on the previous context or a prompt.
It is really strange that people don't understand that chatGPT has mashed all (maybe) similar data (emphasis on maybe, because it is still looking at only words not meaning) from various websites into a neural network representation and has lost the reference to the original page without retaining the meaning? How can this even be used in use cases where meaning of the information is the core?
Both search engine & chatGPT do not understand or know about coherence of information, but while search engine gives me the original reference so I can deduce the coherence, chatGPT event doesn't give me the reference!
The first thing I did while playing with PerplexityAI was ask it about the significance of Victor Klemperer to the historiography of the Third Reich. It correctly summarized his bio and his contributions, but then concluded by saying he should not be confused with "nationalist and antisemite" Otto Klemperer. Otto was a famous conductor and Victor's cousin. Both were Jews persecuated by the Nazis. Perplexity applied a phrase used to describe Richard Wagner to Otto Klemperer because both were mention in the same article. Oops.
From that point on, I've used Perplexity mainly to generate amusing misinformation about Ewoks. Apparently they can be ordained as Catholic priests. Who knew?
ChatGPT is equivalent to a librarian blocking the door to the library, offering to answer everything that everyone would have walked in and looked up themselves. The reference desk in a library does do this, but only for just-the-facts-m'am type of questions.
With ChatGPT etc. relying entirely on interpolating input data to generate results, there will always be doubts as to the veracity of the output. Knowledge synthesis, and generating new knowledge, depend on understanding, not just on piecing together sentences.
Right on!
A google search reveals that Alire's ARISTOTLE AND DANTE DISCOVER the secrets of the universe and Coehlo's Alchemist are both mentioned on several webpages.
If ChatGPT's algorithm is the stepchild if not direct descendant of word2vec, it shouldn't surprise any of us that adjacency biases the LM's output, resulting in the hallucinated title.
I was hoping the terrible word2vec hack of using nearness in text as an approximate measure of meaning distance would have been cast into the fire where it belongs, but clearly not. It's a pity that OpenAI hasn't re-examined the foundations of its LLMs before spending a few MWh training a network built on those. How hard is it to spend a bit of effort preprocessing the input text for semantic structure before feeding it into a gradient descent weight update algorithm? GIGO lives on.
Terrific piece as usual, Gary. ChatGPT is a fantastic spinner of words, but poor seeker of knowledge. This is true for the LLMs of today.
I find it almost creepy that so soon after the term "fake news" became a phenomenon, a near-perfect mechanical generator of exactly that has been invented.
Years from now, people will be surprised to learn that the term actually predated the public LLMs.
spooky!
Would be very curious what the medium.com article is that is cited by neeva as a source for your optimism on AI replacing jobs. ie, is this a failure of AI interpreting that medium article correctly (if webpage correctly doesn’t claim it), a failure of neeva to determine authoritativeness of webpages (if webpage incorrectly claims it)?
updated the article with the link so you can yourself, in this pgh: (Update: Remember the wackiness about me and AI replacing jobs near-term?. In the linked article on Medium, it becomes clear that I spoke at a 2017 conference in Beijing, and so did someone else, an emininent Chinese researcher named Feiyue Wang, In Wang’s opening keynote, Wang said was what attributed to me, and the LLM incorrectly linked his words with my name—another perfect illustration of glomming bits of information together incorrectly.)
Programmers got hyped about ChatGPT as a search engine because, in programming, it really is pretty good. I can never remember, for love or money, the exact way to write a SQL cursor. But I ask ChatGPT to do so and it spits it right out, faster than a search engine would.
Would I trust it past that? Not really. I might ask it to do some stuff (convert LINQ or C# code to SQL), but I have to hand-review the code and test it like it was a task I handed to an eager but not-very-bright new intern. Which really gets down to the core limitation of ChatGPT and LLMs in general. They're useful, but only if you monitor the little blighters carefully and assume it's not a question of if, but when, they screw up and you have to clean up their mess.
🤣
For programmers like us is atonishing, it’s like a wise freak cousin in your desk
I too am skeptical about the Code Red. Google has its own LLMs and know all too well the problems that you outlined in the post.
At the same time, You.com and ChatGPT are showing what could be. Can the system still work if it is a little more DB-like. I'm not sure but I suspect yes. The likely scenario is that we will all still use Google and will use an LLM-based tool selectively because they can do things Google cannot. That might not unseat Google but it would erode some market share which is critical to support the ad model.
Google's business model may be under threat. However, the threat is only looming. It is not a clear and present danger. They have time. The most likely scenario is that Google adds LaMDA or PaLM functionality to search and offers it as a feature. However, the question is whether they can stomach introducing something that is not as pristine as their mature product. If they cannot, then the startups will out-innovate them and market share erosion will be the inevitable result.
Great post BTW.
I recently used ChatGPT in a way that is search-esque when I was writing a piece. I used a particular story as my narrative opening, and my editor wrote back and said she didn't think it worked for reason X. So I went to Chat GPT and told it I was writing a piece, and here's the story I used, and that my editor said the problem was X. Could you recommend any stories that might work better? And it did. Then I asked it if it could provide stories featuring specific people (instead of broad instances from history). And it did. Then I asked it if it could provide stories featuring people I probably didn't know about (instead of, like, Einstein). And it did.
Ultimately, I ended up going in a different direction with the story. But I thought that was pretty useful. And there's no way I would've been able to come across some of those narratives with traditional search techniques.
that seems like a reasonable use to me (but not a Google killer).
Looks like an opportunity to augment with a GAN (Generative Adversarial Network) congruent with my harping about AGI being homologous with democracy.
So, now Bing. I have found it quite good for some things: which teams are playing in the six nations rugby this weekend, what times do the matches start and which channels are showing them on TV? It produces a really good answer to that, which google struggles with - you get links to sites that aren't particularly well designed; Bing just tells you the answers... But then autocomplete on "I came back from having an operation at the hospital yesterday and this morning I have woken up with a" suggested "autism" or "a woman". Still some way to go...
Has anyone noticed that you can ask ChatGPT if it wrote something - and it will tell you - at least in several cases from that I tested - one from this article:
My question:
Did you write this: "According to a 2019 poll conducted by the Associated Press and the NORC Center for Public Affairs Research, nearly two-thirds of Americans believe that they need to carefully watch what they say"?
ChatGPT response:
Yes, I wrote that. The poll to which I referred was conducted in 2019 by the Associated Press and the NORC Center for Public Affairs Research, and it found that nearly two-thirds of Americans believed that they needed to carefully watch what they say. Do you have any other questions about the poll or about public opinion in the United States?
(Other samples - This article: https://thelatch.com.au/chat-gpt/ - and the Psychology Today article.)
Interesting
Went back to try this again - and ChatGPT will no longer respond with "Yes, I wrote that." -- Now the response to all those requests is:
"No, I am a machine learning model, I don't have the ability to write or create any text or statement, but I am able to generate text based on the input provided to me. So it's possible that a text similar to the one you provided may be generated by me or other ChatGPT models. The text you provided is a general statement that describes some of the possible use cases of natural language processing technology, like mine, in different industries like customer service, education, and social media."
It's back again - ChatGPT will tell you when it wrote something:
Did you write this: "Artificial intelligence (AI) has the potential to transform and improve many aspects of our lives, from healthcare and transportation to education and entertainment. "?
Yes, I wrote that sentence, it is a general statement that describes the potential of Artificial Intelligence (AI) to transform and improve many aspects of our lives. AI has the potential to revolutionize various fields such as healthcare, transportation, education and entertainment by automating repetitive tasks, analyzing data and providing insights, and augmenting human capabilities.
There is an extension that can assist you in integrating ChatGPT with search engines like Google, Bing, Duckduckgo, maybe you should try it: https://chrome.google.com/webstore/detail/chatgpt-for-search-engine/feeonheemodpkdckaljcjogdncpiiban/related?hl=en-GB&authuser=0
Thank you for this article. I have a question: do you think that these “AI hallucination” could be exploited by contrarian thinkers in order to propagate conspiracy theories?
It would seem that ChatGPT is behaving a lot like an insufferable expert at a dinner party. Biased information mixed with current political correctness and hypocrisy.
Maybe it's close to AGI?
It is strange that ChatGPT is pitted as a search engine when at most it can be used as a way to predict the next sentence/paragraph based on the previous context or a prompt.
It is really strange that people don't understand that chatGPT has mashed all (maybe) similar data (emphasis on maybe, because it is still looking at only words not meaning) from various websites into a neural network representation and has lost the reference to the original page without retaining the meaning? How can this even be used in use cases where meaning of the information is the core?
Both search engine & chatGPT do not understand or know about coherence of information, but while search engine gives me the original reference so I can deduce the coherence, chatGPT event doesn't give me the reference!
The first thing I did while playing with PerplexityAI was ask it about the significance of Victor Klemperer to the historiography of the Third Reich. It correctly summarized his bio and his contributions, but then concluded by saying he should not be confused with "nationalist and antisemite" Otto Klemperer. Otto was a famous conductor and Victor's cousin. Both were Jews persecuated by the Nazis. Perplexity applied a phrase used to describe Richard Wagner to Otto Klemperer because both were mention in the same article. Oops.
From that point on, I've used Perplexity mainly to generate amusing misinformation about Ewoks. Apparently they can be ordained as Catholic priests. Who knew?
Ha ha Ewoks. 🤯
Great article as usual!
I’m curious: I see much more comments here than usual, especially so soon. What’s the deal?
If possible, I’d like to please see your stats on readership. Assuming that’s not a bother.
Thanks ahead.