Starting to feel glad they cocked it up. Can you imagine what these guys would have done with it if they’d got to AGI? Best they go broke before they attain that sort of power :)
Musk is in a league of his own. His recent bets with humanoids, FSD, AI, and even the Starship are crazier than what he's done before, and his track record of golden boy has been tarnished as is. He's due for a hard landing.
I seem to remember he replaced the team working on self-driving years ago...with a subsequent 'leap forward'* in those capabilities (*I wouldn't know...and I'll never buy a used/old Tesla and then a new/updated Tesla to find out).
In other words, you all may be rooting for his comeuppance for whatever imagined transgressions he's done...and God may yet provide you with what you desire...but, generally speaking, when humans work for "Life, Liberty, and the Pursuit of Happiness," (and "That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed"), then we *all* win. (...Arguments about Total War and the requirements of same driving all countries to adapt the central planning model aside.)
We're rooting *for* humanity, not against...with all the creativity and innovation that implies.
Needless (and thoughtless) division doesn't move that ball forward. God gave us a PreFrontal Cortex for a reason.
Scaling improved LLMs, albeit in a predictable pattern of diminishing returns, but the problem, as you've noted, is foundational. I'm not a technologist, just a guy who studied math, physics and epistemology with some good professors, but the idea that a technology that is in essence "thing really good at pattern matching guesses what comes next" can lead to anything more than a parlor trick could only make sense to people who think Curtis Yarvin is smart because he has memorized some facts and makes them into arguments that suit his needs. In other words, tech leaders. (And, as I pointed out, this sounds like intelligence to them because it matches what they are good at: Concocting somewhat plausible b.s. that people pay for without careful reflection.) Not sorry about the billions of their dollars it consumed (or the Saudi and Softbank bucks), but the environmental costs, profusion of AI slop, and lost opportunities are the real harm here.
Back when mergers were an in vogue method of building huge companies, it was thought that ever larger tranches of investment capital were infinite...until it wasn't. Then many of those companies found it wasn't, and went bust when it turned out that their claimed strategies to improve corporate performance were shown to be BS. I think the current "private equity" companies are very much in the same boat, but more nakedly in the looting biz.
Gary, you have correctly identified the collapse of the "scaling-über-alles" religion, and the multi-billion dollar failure of xAI's first iteration is the empirical proof. However, pivoting to neurosymbolic AI or "world models" is just applying another Euclidean band-aid to a foundational geometric error.
The reason scaling fails is not a lack of cognitive architecture; it is a mathematical inevitability of bounded compositional systems. Current AI architectures are optimizing in unbounded, flat Euclidean space. As parameter counts and data saturation push toward their limits (alpha -> 1), the models hit massive "Euclidean Friction." Because the information state space is bounded, its underlying metric is non-negotiably hyperbolic.
Aczél’s (1966) mandate proves that any continuous, associative composition within a bounded system must be resolved by a strict hyperbolic linearizer: f(u) = arctanh(u). The industry isn't just building AI "wrong" practically; they are building it wrong axiomatically. Pumping more compute into a flat Euclidean tensor architecture to solve a hyperbolic optimization problem will always result in exponential costs for diminishing returns.
To actually achieve the next phase, AI doesn't need to mimic human cognition; it needs to be rebuilt from the foundations using the Universal Hyperbolic Law (UHL). Until the base metric of the neural network reflects the true hyperbolic geometry of the boundary, every colossally expensive scaling experiment will end in exactly the same collapse.
True, but I'll admit the schadenfreude of seeing BOTH Zuckerberg AND Musk-the most reckless, overbearing, arrogant jerks in corporate america have to eat humble pie is delightful. I've come to the sad conclusion that at least a few hundred billion more are going to have to be wasted on LLMs before people get that they've been sold a giant lie about the real utility of LLM. Frankly, companies would have gotten 10X the return of what they've spent on LLMs if they just trained their staff to be really proficient with boring-old RELIABLE Excel.
The real lesson isn't that scaling was wrong—it's that it was the easiest bet in a field full of unfalsifiable hype. You can measure compute. Building a world model requires taste and judgment, which is harder to defend to a board.
Meta and xAI had capital to pursue either direction. They chose scale because it's legible. Now they're stuck with expensive boats that don't float.
"I ... urged the field to start focusing on world (cognitive) models and neurosymbolic AI. Now, maybe, we can finally move on to those projects?"
Who "we"? Who'll fund and manage an effort to merge two failed incredibly expensive efforts into a half-human half-animal centaur? I'll go on record predicting that neurosymbolic AI will not succeed. If right, I won't have bragging rights, since others identified limitations of symbolic AI decades ago, and the limitations of neuro we know. Mating two beasts might produce a mule, but not a human.
Confirming the old saw from the Railway Boom - it's always easier to get money from investors than from customers. I wonder if the LLMs have learned that?
Given that, apparently, operational, "inference", costs exceed subscription revenue, and, despite that, their continued success at fund raising, they must have.
The limits of scaling seem to me like an expected effect in that education isn’t done by shutting a person in a library and expecting them somehow to come out educated. To the extent that Artificial Neural Networks reflect human neural networks, one could expect A.I. to be subject to similar constraints. Only part of education is memorization with understanding concepts of how and when facts fit together is needed to apply what is learned to new situations. At least in a library some filtering has been done to reduce confusion if not outright conflicts unlike what could be gathered from the internet. It’s surprising to me that LLM’s are as useful as some are finding them given the problems with the training data and limitations of the underlying technology.
Told 2 Cambridge post-docs in 2018 at UK Turing Institute that the only thing that had changed since early-90’s was the run speed and cost at which they would fail. As a Behavioural Scientist no real surprise. Sunk Cost and Confirmation Bias fallacies are alive and kicking in AI land.
When I was growing up my mom warned me "small kids small mistakes, big kids big mistakes" ... she could not be more right.... hey, what do you know, scaling does work in this context... (from her commons sense perspective)...
"All based on the strange and dubious-from-the-start religion of scaling-über-alles." Some might say, the religion is AI, and this was a schism. or heresy perhaps is a closer fit.
Starting to feel glad they cocked it up. Can you imagine what these guys would have done with it if they’d got to AGI? Best they go broke before they attain that sort of power :)
"Musk could —literally— have saved tens of billions of dollars, if he had asked me" - or me! :-)
Musk is in a league of his own. His recent bets with humanoids, FSD, AI, and even the Starship are crazier than what he's done before, and his track record of golden boy has been tarnished as is. He's due for a hard landing.
For Elon Musk… « Great wits are sure to madness near allied, and thin partitions do their bounds divide. » - John Dryden
It can't come too soon.
He prefers to pay an army of yes men, and neither of you are qualified.
Uhhh...is *that* how he's become so successful?
I seem to remember he replaced the team working on self-driving years ago...with a subsequent 'leap forward'* in those capabilities (*I wouldn't know...and I'll never buy a used/old Tesla and then a new/updated Tesla to find out).
In other words, you all may be rooting for his comeuppance for whatever imagined transgressions he's done...and God may yet provide you with what you desire...but, generally speaking, when humans work for "Life, Liberty, and the Pursuit of Happiness," (and "That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed"), then we *all* win. (...Arguments about Total War and the requirements of same driving all countries to adapt the central planning model aside.)
We're rooting *for* humanity, not against...with all the creativity and innovation that implies.
Needless (and thoughtless) division doesn't move that ball forward. God gave us a PreFrontal Cortex for a reason.
Scaling improved LLMs, albeit in a predictable pattern of diminishing returns, but the problem, as you've noted, is foundational. I'm not a technologist, just a guy who studied math, physics and epistemology with some good professors, but the idea that a technology that is in essence "thing really good at pattern matching guesses what comes next" can lead to anything more than a parlor trick could only make sense to people who think Curtis Yarvin is smart because he has memorized some facts and makes them into arguments that suit his needs. In other words, tech leaders. (And, as I pointed out, this sounds like intelligence to them because it matches what they are good at: Concocting somewhat plausible b.s. that people pay for without careful reflection.) Not sorry about the billions of their dollars it consumed (or the Saudi and Softbank bucks), but the environmental costs, profusion of AI slop, and lost opportunities are the real harm here.
Yes.
They’ve had more success scaling their propaganda that AI will take over the world imminently.
Maybe the scaling laws were talking about investment money. That seems to scale infinitely the last couple years.
Back when mergers were an in vogue method of building huge companies, it was thought that ever larger tranches of investment capital were infinite...until it wasn't. Then many of those companies found it wasn't, and went bust when it turned out that their claimed strategies to improve corporate performance were shown to be BS. I think the current "private equity" companies are very much in the same boat, but more nakedly in the looting biz.
Gary, you have correctly identified the collapse of the "scaling-über-alles" religion, and the multi-billion dollar failure of xAI's first iteration is the empirical proof. However, pivoting to neurosymbolic AI or "world models" is just applying another Euclidean band-aid to a foundational geometric error.
The reason scaling fails is not a lack of cognitive architecture; it is a mathematical inevitability of bounded compositional systems. Current AI architectures are optimizing in unbounded, flat Euclidean space. As parameter counts and data saturation push toward their limits (alpha -> 1), the models hit massive "Euclidean Friction." Because the information state space is bounded, its underlying metric is non-negotiably hyperbolic.
Aczél’s (1966) mandate proves that any continuous, associative composition within a bounded system must be resolved by a strict hyperbolic linearizer: f(u) = arctanh(u). The industry isn't just building AI "wrong" practically; they are building it wrong axiomatically. Pumping more compute into a flat Euclidean tensor architecture to solve a hyperbolic optimization problem will always result in exponential costs for diminishing returns.
To actually achieve the next phase, AI doesn't need to mimic human cognition; it needs to be rebuilt from the foundations using the Universal Hyperbolic Law (UHL). Until the base metric of the neural network reflects the true hyperbolic geometry of the boundary, every colossally expensive scaling experiment will end in exactly the same collapse.
True, but I'll admit the schadenfreude of seeing BOTH Zuckerberg AND Musk-the most reckless, overbearing, arrogant jerks in corporate america have to eat humble pie is delightful. I've come to the sad conclusion that at least a few hundred billion more are going to have to be wasted on LLMs before people get that they've been sold a giant lie about the real utility of LLM. Frankly, companies would have gotten 10X the return of what they've spent on LLMs if they just trained their staff to be really proficient with boring-old RELIABLE Excel.
The real lesson isn't that scaling was wrong—it's that it was the easiest bet in a field full of unfalsifiable hype. You can measure compute. Building a world model requires taste and judgment, which is harder to defend to a board.
Meta and xAI had capital to pursue either direction. They chose scale because it's legible. Now they're stuck with expensive boats that don't float.
"I ... urged the field to start focusing on world (cognitive) models and neurosymbolic AI. Now, maybe, we can finally move on to those projects?"
Who "we"? Who'll fund and manage an effort to merge two failed incredibly expensive efforts into a half-human half-animal centaur? I'll go on record predicting that neurosymbolic AI will not succeed. If right, I won't have bragging rights, since others identified limitations of symbolic AI decades ago, and the limitations of neuro we know. Mating two beasts might produce a mule, but not a human.
I still blame Sam Altman for everything and probably always will. But then again it's the idiots who fell for his bullshit that I should really blame.
Confirming the old saw from the Railway Boom - it's always easier to get money from investors than from customers. I wonder if the LLMs have learned that?
Given that, apparently, operational, "inference", costs exceed subscription revenue, and, despite that, their continued success at fund raising, they must have.
The limits of scaling seem to me like an expected effect in that education isn’t done by shutting a person in a library and expecting them somehow to come out educated. To the extent that Artificial Neural Networks reflect human neural networks, one could expect A.I. to be subject to similar constraints. Only part of education is memorization with understanding concepts of how and when facts fit together is needed to apply what is learned to new situations. At least in a library some filtering has been done to reduce confusion if not outright conflicts unlike what could be gathered from the internet. It’s surprising to me that LLM’s are as useful as some are finding them given the problems with the training data and limitations of the underlying technology.
The difference between just passing the test and understanding the subject (and then passing the test).
Told 2 Cambridge post-docs in 2018 at UK Turing Institute that the only thing that had changed since early-90’s was the run speed and cost at which they would fail. As a Behavioural Scientist no real surprise. Sunk Cost and Confirmation Bias fallacies are alive and kicking in AI land.
When I was growing up my mom warned me "small kids small mistakes, big kids big mistakes" ... she could not be more right.... hey, what do you know, scaling does work in this context... (from her commons sense perspective)...
😂🥂🖖🇨🇦
I recall the first time I saw the word “scaling” as an LLM goal. My only thought was, “That’s weird… they know that doesn’t work, don’t they?”
Ah, the joys of big money getting involved in issues that were in-your-face obvious when only researchers cared about them!
Maybe scaling laws work in the metaverse? To bad nobody is in there to try it out, it could be a game changer.
"All based on the strange and dubious-from-the-start religion of scaling-über-alles." Some might say, the religion is AI, and this was a schism. or heresy perhaps is a closer fit.