There is an infantile assumption in anything to do with AI that exponential growth is expected as normal, and also sustainable. I wonder why? Is it just a lack of numeracy skills, or basic scientific ignorance? Exponential growth is very rare in the natural world. Thing saturate or break very quickly. I wouldn't be surprised if we only get minor incremental improvements in GPTs from now on, improvements that would be hard to financially justify.
Well, if GPT-5 goes bust, so what? Since when did millennialists stop believing in the end of the world simply because it refuses to come when they predict it? It seems to me that faith in AGI-via-scaling is in the same category of beliefs. But there's always the possibility that Microsoft won't be willing to go to the brink of bankruptcy to test this particular belief.
Building AGI is really hard, yes. Simply scaling up won't be enough, indeed.
Most likely GPT-5 and Google's Gemini will be incremental improvements.
It is clear by now that LLM is not good at accuracy, verification, search, math. What it is, however, is a flexible and versatile approach at generating language and doing language tasks.
Likely OpenAI and Google will build an agent that will classify the input task, prepare background information, run one or more expert LLM, get the results, validate against a vast knowledge base, for some tasks generate and run code in the background, etc.
It is likely that Google will overtake OpenAI as Google has deeper and more diverse technical expertise.
There is also the matter of data. For one thing, copyright issues have become thornier since GPT-4. For another, Generative AI has *already* poisoned the web. Finding good training data has become much, much harder.
Also, training (or perhaps running) a larger model might require resources that are not just expensive but simply inexistent. Even Microsoft Azure cannot provision an infinite number of GPUs, or infinite disk space.
I doubt GPT-5 makes sense financially. GPT-4 hasn't found a market yet beyond curiosity - which is probably fading fast. Language modelling has a great future but a lot of work needs to be done before the technology is really "market ready" for serious applications.
"People might even start to realize that building truly reliable, trustworthy AI is actually really, really hard." I would agree with that. If you take short cuts, they come back to bite you. Size alone is not the problem - we have 100 billion neurons at our beck and call (and have a four pieces limit), but if you don't know what meaning is, and hope to do better by more statistics, the answers are likely to get worse. All this stuff about the new release will fix all the problems - how do you fix a generative AI that gets the wrong answer (and gets stroppy about it)?
In the hopefully-soon-to-be-released video of my talk at EACBPM 2023, there is an example from the GPT3 paper (the last real paper by OpenAI) where, if you look at the quality improvement as a function of model size (and they use a logarithmic x-axis, which is always a warning sign), a quick and dirty calculation tells you that to get to the level of a human on that particular test, the model needs to be something like 10,000 to 100,000 *times* as large... There probably simply is no business case to grow more, and their attempts to scale back (I guess some should be possible) has failed (but would probably never have reached that order of magnitude anyway).
The next AI paradigm will replace similarities with differences and statistics with comparisons and filtering.
In 1973, Allen Newell mentioned, "You can’t play 20 questions with nature and win". But the game “20 Questions” perfectly illustrates how intelligence works. Note the underlying logarithmic complexity of the process. The real-time pressures of most real-world scenarios justify the adoption of this algorithm. It is time to abandon guessing and adopt the 20-question approach to AI.
Meh, kinda feels like the moment Intel discovered you couldnt simply run a core at 5 or 6Ghz. So they took a different path. I think LLMs "training" is probably where the most work needs to be done.
The future of LLMs does not look very bright in my opinion. I predict that LLMs will become obsolete as soon as AGI arrives on the scene.
Instead of gigantic, expensive, know-it-all systems that can't tell the difference between truth and falsehood, or good and bad, we will have many smaller embodied systems. Each will have been raised in the world to acquire a specific type expertise. The systems that I envision will develop an experiential understanding of the world based on visual, tactile and auditory sensors, complemented by a full range of motor effectors. After an initial upbringing, they will develop their expertise pretty much the way humans do, i.e., by going through a specific training or schooling regimen. True language understanding can only be built on top of a perceptual and behavioral framework immersed in the world.
I just don't expect this kind of AI to come from the generative AI paradigm.
There is an infantile assumption in anything to do with AI that exponential growth is expected as normal, and also sustainable. I wonder why? Is it just a lack of numeracy skills, or basic scientific ignorance? Exponential growth is very rare in the natural world. Thing saturate or break very quickly. I wouldn't be surprised if we only get minor incremental improvements in GPTs from now on, improvements that would be hard to financially justify.
The main obstacle is the fundamental impossibility of eliminating shortcomings by increasing the size.
Well, if GPT-5 goes bust, so what? Since when did millennialists stop believing in the end of the world simply because it refuses to come when they predict it? It seems to me that faith in AGI-via-scaling is in the same category of beliefs. But there's always the possibility that Microsoft won't be willing to go to the brink of bankruptcy to test this particular belief.
Building AGI is really hard, yes. Simply scaling up won't be enough, indeed.
Most likely GPT-5 and Google's Gemini will be incremental improvements.
It is clear by now that LLM is not good at accuracy, verification, search, math. What it is, however, is a flexible and versatile approach at generating language and doing language tasks.
Likely OpenAI and Google will build an agent that will classify the input task, prepare background information, run one or more expert LLM, get the results, validate against a vast knowledge base, for some tasks generate and run code in the background, etc.
It is likely that Google will overtake OpenAI as Google has deeper and more diverse technical expertise.
There is also the matter of data. For one thing, copyright issues have become thornier since GPT-4. For another, Generative AI has *already* poisoned the web. Finding good training data has become much, much harder.
Also, training (or perhaps running) a larger model might require resources that are not just expensive but simply inexistent. Even Microsoft Azure cannot provision an infinite number of GPUs, or infinite disk space.
I doubt GPT-5 makes sense financially. GPT-4 hasn't found a market yet beyond curiosity - which is probably fading fast. Language modelling has a great future but a lot of work needs to be done before the technology is really "market ready" for serious applications.
"People might even start to realize that building truly reliable, trustworthy AI is actually really, really hard." I would agree with that. If you take short cuts, they come back to bite you. Size alone is not the problem - we have 100 billion neurons at our beck and call (and have a four pieces limit), but if you don't know what meaning is, and hope to do better by more statistics, the answers are likely to get worse. All this stuff about the new release will fix all the problems - how do you fix a generative AI that gets the wrong answer (and gets stroppy about it)?
In the hopefully-soon-to-be-released video of my talk at EACBPM 2023, there is an example from the GPT3 paper (the last real paper by OpenAI) where, if you look at the quality improvement as a function of model size (and they use a logarithmic x-axis, which is always a warning sign), a quick and dirty calculation tells you that to get to the level of a human on that particular test, the model needs to be something like 10,000 to 100,000 *times* as large... There probably simply is no business case to grow more, and their attempts to scale back (I guess some should be possible) has failed (but would probably never have reached that order of magnitude anyway).
The next AI paradigm will replace similarities with differences and statistics with comparisons and filtering.
In 1973, Allen Newell mentioned, "You can’t play 20 questions with nature and win". But the game “20 Questions” perfectly illustrates how intelligence works. Note the underlying logarithmic complexity of the process. The real-time pressures of most real-world scenarios justify the adoption of this algorithm. It is time to abandon guessing and adopt the 20-question approach to AI.
https://alexandernaumenko.substack.com/
Scaling laws eventually set into any enterprise. No surprise there..
Meh, kinda feels like the moment Intel discovered you couldnt simply run a core at 5 or 6Ghz. So they took a different path. I think LLMs "training" is probably where the most work needs to be done.
The future of LLMs does not look very bright in my opinion. I predict that LLMs will become obsolete as soon as AGI arrives on the scene.
Instead of gigantic, expensive, know-it-all systems that can't tell the difference between truth and falsehood, or good and bad, we will have many smaller embodied systems. Each will have been raised in the world to acquire a specific type expertise. The systems that I envision will develop an experiential understanding of the world based on visual, tactile and auditory sensors, complemented by a full range of motor effectors. After an initial upbringing, they will develop their expertise pretty much the way humans do, i.e., by going through a specific training or schooling regimen. True language understanding can only be built on top of a perceptual and behavioral framework immersed in the world.
I just don't expect this kind of AI to come from the generative AI paradigm.