I’m not sure that differentiating poetic AGI from “mere” AGI helps. Surely AGI is a state that you either achieve or you don’t. Personally I don’t think we’re even close to it or that we ever will achieve it. Our understanding of the brain is so limited and constrained by the things we do understand. It’s only 100 years or so since we thought the brain worked like a telephone exchange, then 50 years ago it was a computer.
I’ve spent my entire 40+ years working in tech but I don’t believe any of this. It’s arrogance and hubris beyond belief to think that this is even remotely achievable.
We can never achieve AGI or ASI or call it what you will. Merely checking a bunch of boxes denoting random aspects of human thought to survive another day on the planet does not equal intelligence.
As far as Kurzweil's litmus test using poetry, he needs to know that we already had AGI had 2020 or even before.
These days we have a load of "free verse" garbage masquerading as poetry. Take Rupi Kaur who is considered one of the best poets of her generation. Her "poems" feels like a random sentence idea coming into somebody’s head who never tries to raise questions for that very thought but is more obsessed with jotting it down using breaks in whatever randomness it entails.
Here is one of Kaur's "poems" called "growth":
You do not just wake up
And become a butterfly
That's it. That's the poem. On that score LLMs are ideal for poetry!
Likewise, in both experience and attitude. The ironic thing is that the truly great minds, people like Feynman, feel a sense of real humility when confronting the universe.
I think the tests for AGI are at very best incomplete. Were I to ever waste my money with one of the bots, I'd start asking questions about how it felt about its mother, then its grandmother, and so on, until I drove it into a state of hallucination. In other words, exposing the inability of AI to simply say, "I don't know" or "I don't remember". (But perhaps I am wrong.)
Thanks for the link. Very entertaining and thought provoking (as always with RPF), though dated in some respects. Abu Mustafa used the bird-plane analogy in his ML class, wonder if he got it from Feynman.
that essay breaks my heart since i wrote a piece in 2020 with Ernest Davis called GPT Bullshit artist and the editor changed the title to GPT Bloviator under heavy protest.
Sincere apologies for breaking your heart...but on the good side: at least some progress toward truth has been made! ;-)
I haven't read your article Gary (is available somewhere?) so apologies again, but I found the reliance on 'On Bullshit' (Frankfurt) goes a long way toward debunking the LLM -> AGI assumption implied by the use of 'hallucinations' (a deceiving psychological metaphor...bullshit in the Frankfurtian sense). With the bullshit flying around in the political media these days...maybe the time is right for 'ChatGPT (really LLMs) is Bullshit' to take hold. Just pining optimistically for the public push for AI safety/responsibility.
I sat next to Ray at a lunch for Gary Kasparov at Google. There were maybe 12 other top Google luminaries in AI there (I'm not one; I just introduced his talk).
He took up easily a third of all the airtime, if not half. A blowhard, in other words.
One problem with Kurzweil's prediction is that there are no signs to check for between 'now' and his professed dates. So it is fully non-falsifiable until the very end. In that sense it is a kind of 'messianic' prediction, or, a prophecy.
Kurzweil should answer the question what kind of advances he expects to see shortly before his 2029 date, say in 2026. Not full AGI, sure, but what are his telltale signs that corroborate his prophecy? Being more of a messianic prophet, he won't be able to play ball, I suspect.
My work for the blind has been compared to Ray Kurzweil's reader for the blind.
This was from an NFB state leader who had no idea Ray was my programming hero.
His relentlessness in programming to recognize hundreds of fonts is what I admire.
His belief in AGI has always mystified me.
Not ever happening unless we truly see God's hand touching a factory full of GPUs.
AGI is not inherent in machines.
Human I is based on a complex brain that infers, deduces, dreams, fantasizes, expands with some chemicals and meditations, thinks outside the box, and is intimately entwined with a life force.
ML - not so much - not so much at all.
HIs are stranger than AIs. The eight billion running around right now are trillions and gazillions times more interesting than any gen AI.
AI - boring.
AGI - boring and will never exist until God wants to make a point.
Are you sure? Maybe a life force is a field to be modeled like Maxwell modeled electromagnetic forces. Maybe dreams are reality and thus the basis for a competent world model. Why assume that any silicon machine could possibly have a competent world model. AIs have such a limited concept of existence.
AIs seem to be several levels of complexity and reality away from human awareness as I experience it. And what I experience is probably levels of complexity removed from what we may eventually know.
Because an AI arranges words based on smatterings of utterances (maybe a trillionth of unique human utterances since 2001), there's not much there there in the largest model.
Next degree of separation is that words are like Plato's shadows on the cave wall except less definite. Except in the hard sciences they hardly represent a competent world model. Given our inability to reconcile some basic physics phenomena, humans may never have a competent world model which tells me that AIs won't either.
Having approached the limits of Moore's Law and electrical power generation, why pretend we can "grow" forever in the AI space any more than we can grow forever in the fossil fuel space.
Cory Doctorow nailed AIs potential by comparing it to other science fiction fantasies - just because you can dream it doesn't make it possible.
Thanks, Gary very helpful. It would be nice if we actually understood what intelligence was in the first place, so we know what we’re trying to achieve.
Ray Kurzweil is undoubtedly one of the brightest minds in the world when it comes to computing, but let's think through what it would take for his prophecy to work out.
Achieving artificial general intelligence, a computer doing any cognitive (not motor) task that a human can do, would first require a reasonable catalog of just what a human can do. I don't think that there is much prospect that we could compile such a catalog in 4.5 years, even if we had a Manhattan Project worth of effort.
Further, we would need to assess the validity of any tests we could come up with. Researchers are currently claiming that LLMs (large language models) possess all sorts of cognitive skills based on very poor experimental methodology.
The hypothesis is that a language model that passes a test associated with a skill has that skill, but it could pass the test based the alternative hypothesis that it has memorized language patterns. As the number of sentences used in training increases, the probability increases that the model has memorized the response it produces rather than mastered the skill. An actor can recite the lines of a mathematical genius without being one, for example. For a large language model to suddenly undergo a phase transition from word guess to general cognition would be something analogous to spontaneous generation or a miracle.
Similarly, one needs to demonstrate that a specified test used to assess a cognitive skill is also a valid test of that skill. One can claim that any benchmark is a measure of anything, assessing whether it is, requires a different set of evaluations.
I have written about what I think is needed to achieve artificial general intelligence https://thereader.mitpress.mit.edu/ai-insight-problems-quirks-human-intelligence/ . I don't think that achieving AGI is impossible, but I do think that the tools we have today are not going to do it, and that it is difficult to guess when the required tools will be invented.
I think you are way too soft on Kurzweil. Maybe he knows a lot about AI but he has no clue about human general intelligence when he defines a "mere" AGI as: AI that can perform any cognitive task an educated human can.
For Kurzweils prophecy of AGI to succeed by 2032, he needs to specify AGI with a little help from people who know about human general intelligence. My suggestion:
AGI is AI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can. The task should not be novel; should not require any subjective experience nor theory of mind nor any biographical memory; it should not require to understand the meaning of the task in the real multisensorial world, and it should not require one trial learning, the generalization from a few examples or the transfer of knowledge between totally different domains.
Let me explain:
He does not mention what a cognitive task is. Is this any task where the input and the output is words or numbers? OK, then he might be right. But that is an extremely simplified version of human cognition. He excludes an excellent poet, but he probably also should exclude excellent scientists. Hence we should add that AGI is OK if it's average. He also does not mention that the task should be performed correctly. This is a big challenge since hallucinations are a core feature of the generative AI's we have today.
AGI is AI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can.
I'd need a whole page to list all the other areas of human cognition that AI-sciences didn't even touch today, for which we need Einsteinian revolutions in the AI sciences. Since there is no indication whatsoever of slightest move in that direction, we better add them to the equation too if we want to succeed with AGI. A few examples:
Human cognition is profoundly rooted in a physical and sociocultural reality. We need a revolutionary breakthrough to invent AI systems that understand the meaning of their inputs and outputs in the real multisensorial world.
We also need a revolution for AI systems to perform totally novel cognitive tasks or tasks that need to be performed in a context that was not part of the training data. Hence the task should be non-novel or routine.
Given the importance for human cognition of personal biographical memory, common sense, subjective experiences (eg. feelings and showing affections), the awareness of the ideas and feelings of others (Theory of Mind) and meta-cognition. It's safer to exclude these of the definition, because that will require another major AI breakthrough.
Human cognition is also lot about one-trial-learning like (like most animals can), generalization from a few example and transferring knowledge or skills from one domain to another: from sports to business, from chemistry to cooking, from geometry to art, from astronomy to nuclear physics… and vice versa. In daily life: using a plastic bottle as a funnel, a ceramic mug to sharp a knife, a folded cloth hanger to temporarily replace the broken stand for my laptop.
Hence, for Kurzweils prophecy of AGI to succeed by 2032, he needs to specify it with a little help from people who know about human general intelligence. My suggestion:
By 2032 we might have AGI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can. The task should not be novel; should not need any subjective experience nor theory of mind nor any biographical memory; it should not require to understand the meaning of the task in the real multisensorial world, and it should not necessitate generalization from a few examples nor the transfer of knowledge between totally different domains, nor one-trial-learning
You know it was actually Ray Kurzweil who first pointed out how ridiculous “AGI“ was, because artificial intelligence was always supposed to be general! As opposed to what?! We need a fundamental understanding of what we’re doing and right now we’re just making a lot of tech and using a lot of energy.
What's the difference between AGI and a big-ass self-updating database that's big enough to encompass sufficient knowledge and is powered by sufficient compute to pass as AGI for most people?
Kurzweil is a one trick pony. He identifies trends from the past, in a certain way, and then asserts that the future has to follow the pattern. I find him just a harmless old fool
Kudos for keeping it real and sharing the updates and changes as they happen.
I’m here for exactly this.
Hi Gary
I’m not sure that differentiating poetic AGI from “mere” AGI helps. Surely AGI is a state that you either achieve or you don’t. Personally I don’t think we’re even close to it or that we ever will achieve it. Our understanding of the brain is so limited and constrained by the things we do understand. It’s only 100 years or so since we thought the brain worked like a telephone exchange, then 50 years ago it was a computer.
I’ve spent my entire 40+ years working in tech but I don’t believe any of this. It’s arrogance and hubris beyond belief to think that this is even remotely achievable.
Keep up your skepticism!
We can never achieve AGI or ASI or call it what you will. Merely checking a bunch of boxes denoting random aspects of human thought to survive another day on the planet does not equal intelligence.
As far as Kurzweil's litmus test using poetry, he needs to know that we already had AGI had 2020 or even before.
These days we have a load of "free verse" garbage masquerading as poetry. Take Rupi Kaur who is considered one of the best poets of her generation. Her "poems" feels like a random sentence idea coming into somebody’s head who never tries to raise questions for that very thought but is more obsessed with jotting it down using breaks in whatever randomness it entails.
Here is one of Kaur's "poems" called "growth":
You do not just wake up
And become a butterfly
That's it. That's the poem. On that score LLMs are ideal for poetry!
Likewise, in both experience and attitude. The ironic thing is that the truly great minds, people like Feynman, feel a sense of real humility when confronting the universe.
I think the tests for AGI are at very best incomplete. Were I to ever waste my money with one of the bots, I'd start asking questions about how it felt about its mother, then its grandmother, and so on, until I drove it into a state of hallucination. In other words, exposing the inability of AI to simply say, "I don't know" or "I don't remember". (But perhaps I am wrong.)
+1 re Feynman. He did say this: https://www.youtube.com/watch?v=ipRvjS7q1DI
Thanks for the link. Very entertaining and thought provoking (as always with RPF), though dated in some respects. Abu Mustafa used the bird-plane analogy in his ML class, wonder if he got it from Feynman.
Fyi: Chatgpt is Bullshit (in the soft sense at least)
https://link.springer.com/article/10.1007/s10676-024-09775-5
that essay breaks my heart since i wrote a piece in 2020 with Ernest Davis called GPT Bullshit artist and the editor changed the title to GPT Bloviator under heavy protest.
Sincere apologies for breaking your heart...but on the good side: at least some progress toward truth has been made! ;-)
I haven't read your article Gary (is available somewhere?) so apologies again, but I found the reliance on 'On Bullshit' (Frankfurt) goes a long way toward debunking the LLM -> AGI assumption implied by the use of 'hallucinations' (a deceiving psychological metaphor...bullshit in the Frankfurtian sense). With the bullshit flying around in the political media these days...maybe the time is right for 'ChatGPT (really LLMs) is Bullshit' to take hold. Just pining optimistically for the public push for AI safety/responsibility.
I also like philosopher Luke Stark's "ChatGPT is Mickey Mouse".
I sat next to Ray at a lunch for Gary Kasparov at Google. There were maybe 12 other top Google luminaries in AI there (I'm not one; I just introduced his talk).
He took up easily a third of all the airtime, if not half. A blowhard, in other words.
To drown out Gary is no mean feat :^9
Ah, my bad, sorry. I meant his questions & Gary's ANSWERS took up 1/3 - 1/2 the air time.
:^) ok .. nutritious "meal"
One problem with Kurzweil's prediction is that there are no signs to check for between 'now' and his professed dates. So it is fully non-falsifiable until the very end. In that sense it is a kind of 'messianic' prediction, or, a prophecy.
Kurzweil should answer the question what kind of advances he expects to see shortly before his 2029 date, say in 2026. Not full AGI, sure, but what are his telltale signs that corroborate his prophecy? Being more of a messianic prophet, he won't be able to play ball, I suspect.
But he doesn't adapt his end date? Interesting.
My work for the blind has been compared to Ray Kurzweil's reader for the blind.
This was from an NFB state leader who had no idea Ray was my programming hero.
His relentlessness in programming to recognize hundreds of fonts is what I admire.
His belief in AGI has always mystified me.
Not ever happening unless we truly see God's hand touching a factory full of GPUs.
AGI is not inherent in machines.
Human I is based on a complex brain that infers, deduces, dreams, fantasizes, expands with some chemicals and meditations, thinks outside the box, and is intimately entwined with a life force.
ML - not so much - not so much at all.
HIs are stranger than AIs. The eight billion running around right now are trillions and gazillions times more interesting than any gen AI.
AI - boring.
AGI - boring and will never exist until God wants to make a point.
Are you sure? Maybe a life force is a field to be modeled like Maxwell modeled electromagnetic forces. Maybe dreams are reality and thus the basis for a competent world model. Why assume that any silicon machine could possibly have a competent world model. AIs have such a limited concept of existence.
AIs seem to be several levels of complexity and reality away from human awareness as I experience it. And what I experience is probably levels of complexity removed from what we may eventually know.
Because an AI arranges words based on smatterings of utterances (maybe a trillionth of unique human utterances since 2001), there's not much there there in the largest model.
Next degree of separation is that words are like Plato's shadows on the cave wall except less definite. Except in the hard sciences they hardly represent a competent world model. Given our inability to reconcile some basic physics phenomena, humans may never have a competent world model which tells me that AIs won't either.
Having approached the limits of Moore's Law and electrical power generation, why pretend we can "grow" forever in the AI space any more than we can grow forever in the fossil fuel space.
Cory Doctorow nailed AIs potential by comparing it to other science fiction fantasies - just because you can dream it doesn't make it possible.
Hey Andy, thanks for chatting.
Not sure about the fundamental limitations but I'll think on it.
Re-revising the Gloat Date on my calendar. January 1st, 2030 it is! 📆 😏
Thanks, Gary very helpful. It would be nice if we actually understood what intelligence was in the first place, so we know what we’re trying to achieve.
Ray Kurzweil is undoubtedly one of the brightest minds in the world when it comes to computing, but let's think through what it would take for his prophecy to work out.
Achieving artificial general intelligence, a computer doing any cognitive (not motor) task that a human can do, would first require a reasonable catalog of just what a human can do. I don't think that there is much prospect that we could compile such a catalog in 4.5 years, even if we had a Manhattan Project worth of effort.
Further, we would need to assess the validity of any tests we could come up with. Researchers are currently claiming that LLMs (large language models) possess all sorts of cognitive skills based on very poor experimental methodology.
The hypothesis is that a language model that passes a test associated with a skill has that skill, but it could pass the test based the alternative hypothesis that it has memorized language patterns. As the number of sentences used in training increases, the probability increases that the model has memorized the response it produces rather than mastered the skill. An actor can recite the lines of a mathematical genius without being one, for example. For a large language model to suddenly undergo a phase transition from word guess to general cognition would be something analogous to spontaneous generation or a miracle.
Similarly, one needs to demonstrate that a specified test used to assess a cognitive skill is also a valid test of that skill. One can claim that any benchmark is a measure of anything, assessing whether it is, requires a different set of evaluations.
I have written about what I think is needed to achieve artificial general intelligence https://thereader.mitpress.mit.edu/ai-insight-problems-quirks-human-intelligence/ . I don't think that achieving AGI is impossible, but I do think that the tools we have today are not going to do it, and that it is difficult to guess when the required tools will be invented.
Herbert, do you have an update to your 2021 book, or is it still appropriate in spite of the present progresses?
I think you are way too soft on Kurzweil. Maybe he knows a lot about AI but he has no clue about human general intelligence when he defines a "mere" AGI as: AI that can perform any cognitive task an educated human can.
For Kurzweils prophecy of AGI to succeed by 2032, he needs to specify AGI with a little help from people who know about human general intelligence. My suggestion:
AGI is AI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can. The task should not be novel; should not require any subjective experience nor theory of mind nor any biographical memory; it should not require to understand the meaning of the task in the real multisensorial world, and it should not require one trial learning, the generalization from a few examples or the transfer of knowledge between totally different domains.
Let me explain:
He does not mention what a cognitive task is. Is this any task where the input and the output is words or numbers? OK, then he might be right. But that is an extremely simplified version of human cognition. He excludes an excellent poet, but he probably also should exclude excellent scientists. Hence we should add that AGI is OK if it's average. He also does not mention that the task should be performed correctly. This is a big challenge since hallucinations are a core feature of the generative AI's we have today.
AGI is AI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can.
I'd need a whole page to list all the other areas of human cognition that AI-sciences didn't even touch today, for which we need Einsteinian revolutions in the AI sciences. Since there is no indication whatsoever of slightest move in that direction, we better add them to the equation too if we want to succeed with AGI. A few examples:
Human cognition is profoundly rooted in a physical and sociocultural reality. We need a revolutionary breakthrough to invent AI systems that understand the meaning of their inputs and outputs in the real multisensorial world.
We also need a revolution for AI systems to perform totally novel cognitive tasks or tasks that need to be performed in a context that was not part of the training data. Hence the task should be non-novel or routine.
Given the importance for human cognition of personal biographical memory, common sense, subjective experiences (eg. feelings and showing affections), the awareness of the ideas and feelings of others (Theory of Mind) and meta-cognition. It's safer to exclude these of the definition, because that will require another major AI breakthrough.
Human cognition is also lot about one-trial-learning like (like most animals can), generalization from a few example and transferring knowledge or skills from one domain to another: from sports to business, from chemistry to cooking, from geometry to art, from astronomy to nuclear physics… and vice versa. In daily life: using a plastic bottle as a funnel, a ceramic mug to sharp a knife, a folded cloth hanger to temporarily replace the broken stand for my laptop.
Hence, for Kurzweils prophecy of AGI to succeed by 2032, he needs to specify it with a little help from people who know about human general intelligence. My suggestion:
By 2032 we might have AGI that can correctly perform any linguistic and mathematical routine cognitive task, an average educated human can. The task should not be novel; should not need any subjective experience nor theory of mind nor any biographical memory; it should not require to understand the meaning of the task in the real multisensorial world, and it should not necessitate generalization from a few examples nor the transfer of knowledge between totally different domains, nor one-trial-learning
You know it was actually Ray Kurzweil who first pointed out how ridiculous “AGI“ was, because artificial intelligence was always supposed to be general! As opposed to what?! We need a fundamental understanding of what we’re doing and right now we’re just making a lot of tech and using a lot of energy.
Whatever happens in 2029, Ray will say "Ta Dah! AGI!"
Which Aguera y Arcas and Norvig declared about LLMs in 2023. They beat him to punch by 6 years.
What's the difference between AGI and a big-ass self-updating database that's big enough to encompass sufficient knowledge and is powered by sufficient compute to pass as AGI for most people?
Right. Clever programming with sufficient speed.
Kurzweil is a one trick pony. He identifies trends from the past, in a certain way, and then asserts that the future has to follow the pattern. I find him just a harmless old fool
I'll take Vogon poetry over AI poetry
Perhaps they are the same
AGI is like obscenity.
We’ll know it when we see it.