In his book 'The myth of AI' (2021), Erik Larson argued that it is precisely the pursuit of Big Data(sets) that has been hindering real progress towards AGI. Interesting to see this convergent argument.
Anyone who worked with neural networks could have predicted this. also after reading the comment section, what is it about AI that seems to bring out the crazy pseudophilosophers out in force? I swear AI sphere used to be math nerds arguing about statistical modelling of reality, now its mostly people who failed remedial math yelling about chatGPT and how it solved a riddle they copied from google.
I should have used "no more than" for clarity. My intended point, such as it was, was that you don't get an emergent property by simply making a bigger version of an inherently limited system. Reinforcing what Greg wrote. Thanks for the observation, they always help me clarify my own thoughts :-)
But weren't the scaling laws already saying that the performance increases as a logarithmic function of the data? What is new in this paper exactly? The title sounds like a statement of the scaling laws. I am confused.
I know you’re joking, but even in chaos there is structure. To be able to correctly model an idiot, you will quite likely have to understand the fundamentals of every human, idiot or not.
Ok, so LLMs can't do this, and can't do that. Seems reasonable, nothing in all of reality can do everything. So until further notice, we could talk about what LLMs can do, and how we can put those features to best use.
When you consider how dumb the internet has made the actual people of the world, why does anyone think pouring more and more of the internet into these machines will make them intelligent?
How much can this be improved by changing the structure of the network and the training regimen, though? I know it's not directly comparable, but it seems a little odd that state of the art LLMs have an order of magnitude more parameters than we have neurons in our skulls, and have been trained on vastly more knowledge than any human could ever read, yet are still so relatively dumb.
BTW, in neurobiology, a glial meshwork has a primary role and controls the functioning of a neural network. Similarly, a semantic multilingual model has a central role and uses a statistical multilingual model for word forms generation.
So now we're saying it has to be reliable before we count it as general intelligence? (Nothing against reliability but it isn't what proves a concept, especially a concept such as general intelligence.) I agree with this technical sentiment that LLMs when overfitted, untuned, and unaugmented with other vectors, are messed up to use in general and not intelligent. For some of these LLM wrapper type apps, It's kind of like using a bell curve when a flow chart would make more sense logically. Sure, we totally need more than just LLMs, but that's what AI has been since before the internet. A bunch of non-LLM non-transformer type AI technology that doesn't rely on pretrained models. So we have AGI it's just not superintelligent. We've already done it. I would even summarize your whole argument as "I wish AGI wasn't so general that it's unreliable. Now let's do some real AI stuff besides just pretrained transformers ad nauseum."
Of course, OpenAI, Google, and friends are already aware of this, which is why they are engineering the hell *around* the models, instead of supersizing them or spend another three years on fine-tuning: GPT3 was already 175B in 2019. They fine-tuned it for three years to make it 'helpful, honest, harmless' and after launch in 2022 the jailbreaks were so bad that they had to resort to filtering (an admission of defeat) within months. Much more has been done, but sizing apparently not. https://ea.rna.nl/2024/02/07/the-department-of-engineering-the-hell-out-of-ai/
In his book 'The myth of AI' (2021), Erik Larson argued that it is precisely the pursuit of Big Data(sets) that has been hindering real progress towards AGI. Interesting to see this convergent argument.
Hi Gary are you going to write about Devin? Quite a few people saying its a scam in the last few days.
Anyone who worked with neural networks could have predicted this. also after reading the comment section, what is it about AI that seems to bring out the crazy pseudophilosophers out in force? I swear AI sphere used to be math nerds arguing about statistical modelling of reality, now its mostly people who failed remedial math yelling about chatGPT and how it solved a riddle they copied from google.
I should have used "no more than" for clarity. My intended point, such as it was, was that you don't get an emergent property by simply making a bigger version of an inherently limited system. Reinforcing what Greg wrote. Thanks for the observation, they always help me clarify my own thoughts :-)
But weren't the scaling laws already saying that the performance increases as a logarithmic function of the data? What is new in this paper exactly? The title sounds like a statement of the scaling laws. I am confused.
we all know that AI is still in the cave https://www.forbes.com/sites/forbestechcouncil/2024/03/18/the-ai-cave/
I know you’re joking, but even in chaos there is structure. To be able to correctly model an idiot, you will quite likely have to understand the fundamentals of every human, idiot or not.
Ok, so LLMs can't do this, and can't do that. Seems reasonable, nothing in all of reality can do everything. So until further notice, we could talk about what LLMs can do, and how we can put those features to best use.
When you consider how dumb the internet has made the actual people of the world, why does anyone think pouring more and more of the internet into these machines will make them intelligent?
How much can this be improved by changing the structure of the network and the training regimen, though? I know it's not directly comparable, but it seems a little odd that state of the art LLMs have an order of magnitude more parameters than we have neurons in our skulls, and have been trained on vastly more knowledge than any human could ever read, yet are still so relatively dumb.
BTW, in neurobiology, a glial meshwork has a primary role and controls the functioning of a neural network. Similarly, a semantic multilingual model has a central role and uses a statistical multilingual model for word forms generation.
So now we're saying it has to be reliable before we count it as general intelligence? (Nothing against reliability but it isn't what proves a concept, especially a concept such as general intelligence.) I agree with this technical sentiment that LLMs when overfitted, untuned, and unaugmented with other vectors, are messed up to use in general and not intelligent. For some of these LLM wrapper type apps, It's kind of like using a bell curve when a flow chart would make more sense logically. Sure, we totally need more than just LLMs, but that's what AI has been since before the internet. A bunch of non-LLM non-transformer type AI technology that doesn't rely on pretrained models. So we have AGI it's just not superintelligent. We've already done it. I would even summarize your whole argument as "I wish AGI wasn't so general that it's unreliable. Now let's do some real AI stuff besides just pretrained transformers ad nauseum."
I provided a link above. Just YouTube FSD 12.3 and you'll find dozens of videos showing practically flawless driving.
Of course, OpenAI, Google, and friends are already aware of this, which is why they are engineering the hell *around* the models, instead of supersizing them or spend another three years on fine-tuning: GPT3 was already 175B in 2019. They fine-tuned it for three years to make it 'helpful, honest, harmless' and after launch in 2022 the jailbreaks were so bad that they had to resort to filtering (an admission of defeat) within months. Much more has been done, but sizing apparently not. https://ea.rna.nl/2024/02/07/the-department-of-engineering-the-hell-out-of-ai/
https://ea.rna.nl/2024/02/13/will-sam-altmans-7-trillion-ai-plan-rescue-ai/ (the parameter volume seems to scale even worse)
Supply will never exceed demand! AIs can just hallucinate more "data" and feed upon itself. Then: Singularity! (of shit)