“Scale Is All You Need” is dead
AI’s biggest conference vindicates this newsletter’s longstanding critique of generative AI. What comes next?
Over three and half years ago, I launched a campaign against the wrong-headed hypothesis that scaling would get us all the way to AGI, in my infamous essay Deep Learning is Hitting a Wall (March 2022) and in the first essay of this Substack, entitled The New Science of Alt Intelligence (May 2022).
That campaign nearly cost me my career.
My target was the then-extremely-popular notion that we could achieve general intelligence simply by “scaling” large language models, sometimes referred to by the slogan “scale is all you need”. It was so popular you can get it on a t-shirt:
The idea — which has led us all the way into the apparent AI bubble — was that one could use “massive amounts of data – often derived from human behavior – as a substitute for intelligence.”
I said it would never work.
It didn’t.
And now, at last, the field is starting to realize it.
§
Before I get to the breaking news, a quick reminder of some context, and why this matters so much.
As you may well recall, I said (repeatedly) that they key issue was that these systems would never be reliable enough, and that even with more data they would have trouble with hallucinations, factuality, reasoning, outliers, and generalization.
What is news is that you no longer have to take my word for it.
The machine learning community, which has unwelcomed me since I started vocally critiquing the scaling approach, has finally started to come around.
§
For those who have forgotten how things were, and by way of contrast from what I am about to report, here’s a pretty good encapsulation of the scale is all you need view, which I called scaling-über-alles.
It’s been the dominant view for the last half decade, shaping not only AI research, but investment and even government policy. It is not an exaggeration to say that our world has been shaped around it.
Sam Altman was a huge proponent, too, likening alleged “scaling laws” to laws of the universe.
In a blog earlier this year, Altman claimed that
He bet the entire company on this notion. He was wrong.
§
So here’s the news part: NeurIPS, the biggest conference in the field just met, and it’s clear that more and more major figures have finally realized that the jig is up.
Although not everyone realizes it, many do:
Rich Sutton’s keynote also expressed serious concerns with the current approach:
§
It’s not just academics and researchers, either. Customers, too, are starting to realize the limits of the technology. Remember that MIT study showing that 95% of companies aren’t seeing much RoI in Generative AI? McKinsey, BCG and several others have now found similar results. The return on investment that people dream of is just not there, for most customers.
A recent Substack on using LLMs for coding from Josh Anderson beautifully captures a common pattern of enthusiasm followed by disillusionment:
It all works, until it doesn’t. We desperately need more reliable technology.
§
What’s next?
Historians will someday want to write about how the United States immolated itself on a false belief in scaling, and how the Trump administration got snookered, possibly tanking the economy, and sacrificing the autonomy of US states along the way. Others may write about on Altman’s role in stoking that chaos. Some might ask why so many academics who never believed in scaling in the first place were ignored along the way.
I anticipate a whole slew of articles and books like these:
But let’s leave the sociology and economics and intellectual history for another day.
Instead, let’s focus on what we should do now, scientifically. I firmly believe that the conclusion from my May 2022 critique of scaling still applies:
[Scaling] may well be better than anything else we currently have, but the fact that it still doesn’t really work, even after all the immense investments that have been made in it, should give us pause.
And really, it should lead us back to where the founders of AI started. AI should certainly not be a slavish replica of human intelligence (which after all is flawed in its own ways, saddled with lousy memory and cognitive bias). But it should look to human (and animal cognition) for clues. No, the Wright Brothers didn’t mimic birds, but they learned something from avian flight control. Knowing what to borrow and what not is likely to be more than half the battle.
The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
Better late than never. Let’s bring in the cognitive scientists, and stop fantasizing that data and compute will solve all our problems. The time for neurosymbolic AI and world models and causality is now.
Let a glorious new era of truly interdisciplinary AI begin.
Gary Marcus warned you about the challenges to scaling years before anyone else did. Please consider joining over 90,000 others in subscribing.
People like Musk and Altman should be asked “now that scaling laws are shown to not work the way you claimed they do, and people were able to identify this when you were saying otherwise, how have you changed the way you analyze the field now that you know your framework needs adjusting?”
No one will ask that (type of) question to any of the people who were fully throating scaling as THE solution. God knows how few journalists are left out there. But that’s got to be the question asked of all those people and it must be continued to be asked until they give a real answer
It's taken clear thinking, determination and a lot of courage to face down the LLM 'scalers' and 'hypsters' but I think you have done it Gary! Huge kudos to you and enormous thanks from the world in general for making a fundamental contribution to bringing an end to this madness.