Seems to me Sam Altman is a full-on sociopath and anyone who believes what comes out of his mouth or invests in OpenAI is a sucker.
Thanks for this post. I read Karen Hao's excellent book, which is how I found you, Gary, because you made the most sense to me. Am heartened to see my instincts—and you—have been vindicated.
More people should be reading Karen Hao’s book, Gary’s Substack, and Ed Zitron on the insane financials. The people who trust this technology don’t seem to understand it at all.
Sam Altman is no different than other Tech Bros. Once either of them jumped on a hype wave and got a driving seat, it's like a one-way street. The only choice is to double down, build the hype even more, to suck even more VC dollars.
It's a separate thread, but what's absolutely fascinating to me is how uncritically anything AI is funded by investors.
* OpenAI's $300B valuation as of March 2025 (how far from potential profitability?)
* Thinking Machines Lab's $12B valuation before they even publicly revealed what they're working on
* Thousands of completely unknown startups that got 6-digit funding purely based on the premise of being AI startups
And that all before we fundamentally resolved what's allowed use of training data (from a law perspective).
One would think there would be a reflection that it doesn't look sustainable. But I guess I expect too much from VCs. Or underestimate the power of the combination of herd behaviors and greed.
And far, far too much money. People who have money to waste tend to spend it carelessly, particularly if their primary motive is greed for more money and power. I saw an interview of Thiel where he was asked which AI companies he's investing in; he responded to the effect he was investing in all the major players because he didn't know who would emerge on top. Not many have that kind of money to throw away. Also noticeable that people like Zuckerberg have ceased innovating and instead buy up all the competition and new innovative startups (i.e., Instagram and Whatsapp). They become lazy, greedy, monopolistic, and insulated from reality.
Yes, Altman is only one of them (i.e., Alex Karp, David Sacks, Elon Musk, Mark Andreesen, etc.), but it's not the only choice. My heroes are the ones who are doing the *work* needed to improve the human condition, increase prosperity for all, and take care of our beautiful blue planet. I admire Gary and Karen Hao for pursuing verifiable solutions to those ends in spite of ridicule and/or resistance from deluded trend followers. Let's hear it for a little humility, generosity, and collaboration:
"OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.
"Fostering its further development may be among the best things that companies, researchers, and governments can do. Investors, take note."
"Fostering its further development may be among the best things that companies, researchers, and governments can do. Investors, take note."
Now, believing that would assume that investors do listen to Gary Marcus and trust his judgment rather than default to herd behaviors.
That seems to me like a tall request.
But, admittedly, extensively working with and helping early-stage startups left me rather sour about VCs and their influence on the whole startup ecosystem. So, you can officially call me biased.
And those few heroes who go against the tide will remain unsung. The scale of their impact is simply too insignificant (when compared to hundreds of billions of dollars poured into AI darlings of the day).
Well, at least they were warned by one of their betters and Gary keeps it real for those who prefer to think for themselves.
Have you read "The Value of Everything: Making and Taking in the Global Economy" by Mariana Mazzucato?
"A scathing indictment of our current global financial system, The Value of Everything rigorously scrutinizes the way in which economic value has been accounted and reveals how economic theory has failed to clearly delineate the difference between value creation and value extraction. Mariana Mazzucato argues that the increasingly blurry distinction between the two categories has allowed certain actors in the economy to portray themselves as value creators, while in reality they are just moving around existing value or, even worse, destroying it.
"The book uses case studies—from Silicon Valley to the financial sector to big pharma—to show how the foggy notions of value create confusion between rents and profits, reward extractors and creators, and distort the measurements of growth and GDP. In the process, innovation suffers and inequality rises."
"The unsinkable GP Titanic is preparing to depart. Last chance to make the voyage of a lifetime. All aboard!" -- Captain Altman, addressing VCs on the docks
Gary gets it. 😎. OpenAI is a lab. They aren't a business. Subscriptions are nice but the real money comes in from VCs and SoftBank. chatGPT is not a product, it is a public experiment.
Before you bet your business on OpenAI, make sure you understand.
OpenAI hype is directly proportionate to capex, which is going nutz, and add in the — wait for it — "$500 billion valuation", and there you have it...Altman has to front-run GPT-5, as Anthropic doing more with less.
Maybe the AI bubble will burst before it gets really airborne.
I think we need more people amplifying Jonathan Shedler's observation "Al seems to know everything—until it's a topic where you have firsthand knowledge".
I think that's important to amplify, because I believe one of the biggest traps is when you're tempted to rely on LLM/GPT-based systems to answer questions on topics you don't understand or have little experience in. That adds significant risk to your ability to judge the quality and validity of the output.
It's something I've been saying in my critiques for more than a year now. I find that when I'm exploring ideas on a topic I know well, I more readily spot LLM/GPT-based system errors: then when attempting to adjust my prompts to correct for those, other errors - sometimes increasingly significant - can be produced. On topics I have some experience but lack both technical and practical depth, I still have sufficient empirical knowledge to spot problems: but when inquiring about domains I have minimal exposure to, how do I check for / confirm validity?
As a very experienced software developer, I feel comfortable using outputs of LLMs because I can verify that they work as expected and we can create concrete tests for evaluating code very easily. I keep a tight leash on generated code and test that it does what I expect. All of the code agents fail miserably at complex tasks and require human oversight.
To some extent, I share that long-term software-development experience. My early and then subsequent looks at LLM/GPT-based code-generation tools have been less-than-inspiring for a few reasons.
Possibly the two most-significant points were:
a) an apparent propensity to continuing the tradition of essentially "copy 'n paste" of commonly-used (and previously copied / shared code) that is available in large quantities in open-access code repositories. While that code can often be "good enough" as a starting point, it can propagate problems including poor (yet widely used) practices and outright errors: errors that although they may not surface *within* the context of an isolated single function or operation, can cause problems when that code is used as part of an integrated whole; and the follow-on problem,
b) an apparent inability of LLM/GPT-based tools to reason about a specific code fragment with regard to the integrated whole that code might reasonably be expected to exist within (a mixed-to-poor results when attempting to provide that specificity through prompts / starting context). By this I mean lack of consideration for temporal aspects (e.g. retention of state, multiple calls to the function), use of naming conventions consistently within the integrated system, and even more simply, lack of holistic consideration of error handling (e.g. over-compensating on some code sections or classes of error, and completely overlooking sections or classes). All of these considerations have a fundamental economic impact on code integration and maintenance.
That caused me to ponder a couple of questions:
How much of that "average" output is due to the nature of the code ingested to form the corpus? How much of the worlds truly exceptional, exemplar code is available to LLM's?
and that lead to thinking:
1) One might reasonably imagine the exceptional code is proprietary and hidden from view. If that's the case, do LLM's have access to the best code patterns that the worlds best developers / best systems use, or are generalised LLM corpora stuck with a slop of averages?
2) How much is the overarching context - the *why* or intent of a system - fundamentally intrinsic, important or instructive to the way in which the underlying code is best structured? One illustrative example here might be Domain-Specific Language's (DSL's). If that is a distinguishing character of better software-design, is AI-code generation using a generalised LLM a useful approach? Can generalised LLM/GPT-based tools understand the nuance of context - such as domain - and reliably and consistently use that nuance of specificity in filtering or forming the generated code? If not, does this require the development of specialised corpora for domain-specific, compliance or requirement-specific or even context-specific needs?
Perhaps a generalised LLM/GPT-based system can be useful (time or cost-saving) in brain-storming - exploring technical possibilities, or useful in comparing and exploring different technical options, or in domain-independent technical functions (e.g. compressing a file / image, encrypting / decrypting, etc), however, I don't yet see how the larger-context understanding needed to build good quality complete system is available / achievable with the commonly available, generalised LLM/GPT-based systems we have today.
It's not a domain I'm a specialist in, so I can't really critique the validity specifically here.
However, I suspect that as a generalisation, it's likely that very specialised LLM/GPT-based systems that focus on a limited / targeted specialised domain with constrained inquiry, and a specifically-refined, carefully managed corpus curated by subject-experts will have better success than more generalised tools.
While this blog is great and Gary is great at criticizing the LLM approach, in the end of the day, in a very fundamental way, he actually is no different than Altman.
Basically he keeps on saying “funding my way of doing things is the only way to get to AGI.” Kids who believe in the tooth fairy have a more realistic view of the world. After all, they have hard evidence: experience shows them they get $1 every time they lose a tooth. No one and I mean no one on this planet has even the beginning of an idea of how to build a model that even partially explains human intelligence, consciousness and creativity. Not the slightest clue.
To pretend that AGI is even feasible without knowing anything about how the only existing AGI agents (human minds) operate, is just BS of the worst sort.
Machine “intelligence” is extremely useful in specific domains and should be researched and developed. But if we learn anything from the LLM hype machine it’s this: please STFU already about AGI.
Thank you. I have a huge admiration for Gary, and I just kinda mentally tuck away the occasional nods he gives to what needs to be done to achieve AGI. Hey, I don't need everyone to agree with me to value what they have to say.
But I'm entirely with you on this one. My prediction for the year AGI arrives is "never". Can't prove it, but I'd bet it against the field.
The only insight I agree with the AGI folks on is that intelligence didn't always exist on Earth but now it does and it's brought into being by some kind of physical (I'd prefer biophysical) system. So yeah, perhaps it's not *literaly impossible* for human beings to create machines that have "intelligence" similar to our own. But that's the most that can be said, anything beyond that is wild speculation.
And for perspective, us humans don't even know how to make life from non-life. That's a thing that happened somehow, right? There didn't used to be life on Earth, now there's life on Earth, must be some way to do it! But we don't know how. Seems if we're gonna prognosticate when future scientific discoveries will arrive, humans creating *any kind of living thing whatsoever* out of non-living matter ought to happen prior to humans figuring out how to create phenomena currently only known to exist in complex living things.
But hey, that's just me guessing. We're all just guessing.
| us humans don't even know how to make life from non-life.
| That's a thing that happened somehow, right? There didn't used to be
| life on Earth, now there's life on Earth, must be some way to do it!
That reminded me of the Miller-Urey experiment (1953), where basic inorganic compounds (methane, ammonia, hydrogen, and water) were subjected to electrical sparks (simulating lightning), heat and pressure enabling the formation of amino acids (organic basic building blocks of proteins). The experiment attempted in part to simulate what was believed to be conditions like that of early Earth, to test the hypothesis that organic molecules could form from inorganic compounds.
The subsequent 1961 Nirenberg and Matthaei experiment synthesised basic poly(U) RNA (and used that to synthesise a protein).
So perhaps it's fair to say that we have some established experiments that can create (synthesise) the basic molecular building blocks of life, and can reason about how those basics can occur in situ.
From there we can add some arguably hand-wavey-suppositions about how more complex microorganisms might mutate from those molecules in the presence of radiation - and / or could arrive on earth in frozen water from space rocks - to build a hypothesis of how life on earth.
edit: actually, the synthesis of RNA occurred earlier and is created to Severo Ochoa and collaborators in the 1950s. Nirenberg and Matthaei's work was important for using the synthetic RNA molecule (poly-U) to successfully direct the synthesis of a protein.
There are also alternative theories and experimental findings that suggest RNA could have been synthesized from simpler molecules in a prebiotic environment.
Thanks for the reply and the info. I probably overstated my case. My basic argument is that if we can't create artificial versions of very simple organisms, then there's little reason to speculate about us making an artificial version of a mysterious phenomenon that we only know to exist in organisms vastly more complex than the simple organisms that we don't know how to create artificially.
I also agree it's important to research, ask about and critique the motivations of prominent public voices.
However, I'm not sure I can make the hops and jumps required to arrive at the claim of "funding my way of doing things is the only way to get to AGI".
In Gary, I see (as you note) a strong critique of the current myopic focus on a "hyper-space-race" based on a single pathway, using a single technique / approach, with obvious shortcomings and hard limits, that has largely devolved to a financial / commercial race to the bottom of finest hair-splitting minutia. The predominant focus appears to be on monetising the slop of a constrained (if not dead end) technology, instead of other possibilities such as investing in the alternative broad-research required to travel significantly further.
However, I don't see how Robust.AI (a company Gary has a key stake-holding / interest in) - with it's goal of creating an "off-the-shelf" machine-learning platform for autonomous robots - really qualifies as either AGI or a direct funding-competitor to the current LLM / GPT-based system "media darlings". I think that Robust.AI's focus arguably qualifies more as machine intelligence that "is extremely useful in specific domains and should be researched and developed".
My perception is that Gary is generally a) proposing less-specific / broader exploration: more openness to the exploration of alternatives (to LLM's), and b) a proponent of putting safeguards in place to potect humanity from the unintended and negative effects of AI systems.
Including these points in addition to style, I don't see the clear comparison / similarity with S Altman.
Regarding your claim that "no one on this planet has even the beginning of an idea of how to build a model that even partially explains human intelligence": you may ultimately be essentially correct, however that appears at first glance like an absolutist claim that would require significant knowledge to confirm.
I'd argue that statement as presented might be less true than you think.
If you haven't already, I'd encourage you to take a look at the OpenCog Hyperon work of Ben Goertzel and team, and the various projects utilising that ecosystem. My limited view is that Hyperion offers support for a broad and diverse range of approaches that collectively *may* begin to explore the diversity of reasoning processes humans have available.
OpenCog Hyperon is one project I know of, but I suspect there are likely others.
I'm not an AI expert, nor A Reddit influencer, and I don't even use Twitter, so few people will listen. But for work I create mathematical models of real-world processes and I use Bayesian interference to tune their parameters. Not very different from what is done with LLMs. And what I posted more than a year ago somewhere on Substack was something like this:
The main difference between human (and animal!) intelligence on one side and current AI on the other is that we constantly create and adjust little models of reality that we use to make inferences. That's why our infants can learn new concepts on the basis of such a small sample size. But current AI only uses one humungous gigantic model with billions of parameters. It's no match for the creativity, flexibility and cognitive speed of real brains.
What I see now is that this is now increasingly being recognised as the problem of a lack of 'world models'. I am actually quite upbeat to see that it's going to be a long and winding path towards an automated ability to create world models. Time to really appreciate the depth and ingenuity of 'natural intelligence'!
I'd be a lot more upbeat if this 'long and winding road' wasn't destroying the planet and sucking up unfathomable amounts of money that is desperately needed elsewhere.
Valid comment. My hope is that as soon as investors realise it, they'll stop burning billions on it. Then Trump will surely step in, but even he tends to lose interest at some point.
Also likely though is that AGI will be given up and instead an artificial pseudo "intelligence" will be used to further build out a new system of slavery and surveillance empire backed up by automated weapons systems used against dissenters.
Thank you for comparing this to human intelligence and phrasing it in a way I can understand it. The thought of various, adjusting models of reality pairs nicely with my general hypothesis that in order to have functioning intelligence a being needs to have senses!
I like and sgree with much of what you've said here Wolfgang.
My comments in response:
a) if you take a look at tge scope of what Wolfram Alpha does, I"m not certain the underlying models qualify as an "automated ability to create world models" and I would imagine that it's still relatively large (if not humongous/ gigantic). The focus here in those cases is on specialised domain-specific models that might be argued as somewhat bounded or finite, and thar need minimal tweaking once established.
b) I don't get the sense yet - at least not from public-facing comments - that the focus and hype around RNN systems of this specific LLM/GPT-based type is really b slowing or over. I think there's simply too much inflation and sunk investment for it to end without a significant "long tail" run out for the investment to be salvaged or used.
Your point about auto-generating little fallible models and tuning based on tests / feedback is excellent. Have you (or anyone else) seen any specific AI-based examples of this worth reviewing?
Yes, I think the focus will eventually be on creating more and more domain specific models. At some stage, the experience gained might then eventually lead to the ability to create ad hoc, situation specific models, as we ourselves do on a daily basis.
The current LLM focussed big-data model that tries to short circuit understanding will probably find some kind of justification for its existence, because so much money has been sunk into it. But the fact that only a handful of companies pursue this path is almost a guarantee for its eventual failure.
In nature, progress is the result of widely distributed experience and experimentation. Monocultures are not resilient have the tendency to collapse.
I'm a multi-award winning documentary filmmaker with a four-decade track record in films about technology and society. I have been trying for eight years, first via the MIT Media Lab and now via Cambridge University, to get people to understand that it's embodied intelligence via all our senses, and life in the world, that enables humans actually to understand things. Evolution over millions of years has also made us adaptive, versatile and extremely energy efficient - according to Stuart Russell, by a factor of about a million to one. Yann LeCun himself admits that a domestic cat has more actual intelligence than the most advanced 'AI' model. This hasn't changed in those eight years, yet the money thrown at this technology continues to balloon, with terrifying consequences for the climate and society. (And if it's any consolation, nobody is paying me for my work either).
The businesses making the investments either benefit directly from the hype (Amazon, Nvidia, OpenAI, Google Cloud, Amazon AWS, Microsoft Azure) or have huge existing franchises to defend that seem under threat from AI (Google Search, Microsoft Office, Meta, Apple). The politicians just hope that genAI can get them out of a productivity hole. I didn't expect the hype to last this long but that's a heck of a lot of momentum from those companies. And these tools are fun and remarkable even if they are nothing approaching intelligent.
Exactly. Human intelligence is inseparable from a human body and human experience. Shouldn’t that be sort of obvious? Yet, it seems conveniently ignored or forgotten by some otherwise bright people in pursuit of some technological holy grail…
Yes, exactly. I have noticed many people think AI isn't intelligent because it can't do some behavior that humans do well but if we gave a machine all the senses and experiences humans actually get I hypothesize we might get something resembling intelligence.
Jeff Hawkins has proposed a different way to build an AGI and have open sourced their efforts here:
On a side note I think many readers would support microtransactions to support authors. I wrote the following to get the conversation started since the Substack team has made it evident that advertisements might be coming soon.
Excuse me Brad. Are you affirming that you believe that "if we gave a machine all the senses and experiences humans (get), that "we might get something resembling intelligence"?
Hey Sheila, DM me, I'd love to help connect you with philosophers, cognitive scientists, and research institutes that are sympathetic to this view. I certainly am.
Blake, thanks a lot but I don't need introductions: as Director's Fellow at the MIT Media Lab and now as Advisory Board member of the Minderoo Centre at Cambridge University, over the past eight years I've interviewed/recruited dozens of top people in this area: a few who agree are Neil Lawrence, DeepMind Professor of Computer Science at Cambrudge, Christof Koch who runs AllenAI and Melanie Mitchell, computer science prof at the Santa Fe Institute. And, ironically in retrospect, when I first showed the first cut of the film at MIT, Mustafa Suleyman - then on the board of the Media Lab - told me it was 'really important' and I should 'get it out there as soon as possible'. I completely failed to recognise him so didn't take him up - I don't think he'd be quite so keen now, having thrown in his lot with Microsoft (I first interviewed Bill Gates, incidentally, in 1991 or so when he was still a geek wearing acrylic sweaters and bigging up Encarta, the CD Rom precursor of 'all the world's knowledge). So I don't need introductions! I need money to finish the thing and get it out into the world.
Gotchya — maybe try reaching out to the folks at the Cosmos Institute? They would probably be interested in this kind of project. It could elevate their brand.
Thanks - I will take a look. Alison Gopnik is great. Not so inspired by the men you list; it's interesting to me how (with the obvious exception of people responding on this thread and a few others) very gendered this subject is. All the gung ho techno-positivist-entrepreneurs are men, and many of the most eloquent, erudite and expert sceptics are women; apart from Alison, Timnit Gebru, Meg Mitchell, Emily Bender, Melanie Mitchell, Margaret Wertheim and many others. I remember when I first presented my work at the MIT Media Lab, one of my co-Director's Fellows said to me, with pity in his voice, 'The problem you'll have, Sheila, is that these ideas aren't sexy'. By which I think he meant sexy as in 'send a giant phallus to Mars' Sexy, as opposed to 'engage in the subtle, laborious and complex work of sorting out our own planet' Unsexy. I guess developments in the eight years since have only borne him out.
Ha! Yes — skepticism about ambitigious technological projects is generally not sexy. But the story can easily be made sexy with a couple reframings. E.g. "elites misleading the public to enrich themselves" "techno-feudalist overlords desperately trying to satisfy their cosmic worldview" "triumph of gnosticism on the grandest scale ever seen" "tragic resurgence of man's hubristic attempt to surmount his humanity by mastering it" "technological stagnation and the last gasps of optimism"
The story about the quest for artificial intelligence is much older than the last few years, and the psychological motivations go back millennia. That's the story I would tell — of hubris, desperation for a theory of mind, desire to understand what makes humans different.
The story has to be "here again, we run up against the fundamental confusions in the project of trying to predict, explain, and replicate the complexity of biological life."
I hear you on the gender piece, but I wouldn't get too hung up on it. Framing optimism/pessimism as a gendered thing could hurt your story by needlessly alienating key audiences and making it seem like you have a political bone to pick. Also, don't sleep on Juliet Floyd! Just ask the female philosophers which men they'd recommend. They generally care far less about the gender stuff than about the ideas.
(as, indeed, is Margaret Wertheim's 'Pythagoras' Trousers'.) My film/films are aimed at wide general audiences, so I prefer to frame the argument in cats, cookies, spiders, toddlers, cab drivers, nurses, truckdrivers etc. More jokes in those areas and more familiar 'aha!' moments. But thanks.
The thing that I find most troubling about the launch was just how desperately people cried for a return to the sycophantic style of 4o — because it was more engaging for the companion/therapist/life coach uses that have come to characterize most people's relationships with GenAI. Sam acknowledged that this was sad and gross, but caved anyway and gave them 4o.
This would have all been inconceivable two years ago. I'm really curious how the street will react on Monday.
Thanks for your writing on this. You've earned a victory lap — and a break.
It's how they juice their user/subscriber numbers to try and convince investors that the money is coming. It's extremely gross and sad and exploitative. Watching people have full on mental breakdowns because o4 got shut down made me feel dirty and quite angry that it was allowed to happen.
I expect a lot of people inside OpenAI feel the same way. There has to be growing disillusionment. They thought they’d unlock human potential and it turns out they’re just burying us deeper in the solipsism of our screens.
In 1957-59 I studied numerical methods under T. S. Motzkin; I was an undergraduate but at his invitation was able to participate in his graduate seminar. I was also very interested in the foundations of mathematics and formal logic. This was in the immediate aftermath of the Dartmouth Workshop and everyone I knew was very excited about "artificial intelligence." But based on my own knowledge of mathematics and computers and after some discussions with Motzkin and others I concluded that AI was going to remain a very limited model of what is generally understood by the word "intelligence" unless or until some very profound and unforeseen discoveries were to come about. As wave after wave of enthusiasm for AI has risen in the intervening 78 years I have scanned each for evidence of fundamental transformation but never finding any. Nothing has ever been done that could not be analyzed in the terms known to T.S. Motzin and other first-class mathematicians of his era. We long ago reached the limits of what is possible by engineering tweaks to AI programs. Anyone who truly wishes to pursue AGI, it seems to me, must seek to better understand just how the brain creates the mind.
This is a fundamental arrogance of people in the AI space. We don't fully understand the brain or the mind yet but have the hubris to believe we can create the equivalent. Having a paper confirm that it is just an imitation and not the real thing is nice but it's always just been an imitation. These tools try to mimic the output that comes from humans but do nothing to emulate how a brain actually works despite using terms like "neural networks."
I think this hits the nail on the head. The common thinking is that AGI = human intelligence. Ask any group of Neuroscientist where we are in the understanding of human intelligence you’ll see a lot of shrugged shoulders. Without a good model of the human brain/mind, the best thing you can do is the equivalent of throwing paint against a wall trying to get an exact replica of the Mona Lisa with nothing more than a stick figure drawing for reference.
There have been several state sponsored programs going on in various counties for the last 10 to 20 years. The US has one, China has one. The EU shut down the Human Brain project in 2023. They learned a lot, but any goal of a complete understanding of the human brain was not achieved.
I think we have a long way to go before we have a deep enough understanding of the brain/mind to create human equivalent AGI.
As formulated originally by Newell and Simon, their physical symbol system hypothesis (PSSH) is altogether independent of the mechanics of implementation. As they say in their landmark 1976 ACM paper:
"The Physical Symbol System Hypothesis. A physical symbol system has the necessary and sufficient means for general intelligent action." Wikipedia has a useful brief summary.
Thanks. The Wikipedia article suggests that PSSH, at least in its stronger form, excludes neural networks. I don’t have a dog in this fight, so happy to include neural networks under the umbrella if that makes sense. I certainly think about LLMs symbolically and am confident they can be understood this way.
I don’t think you can attribute any sort of symbolic functionality to LLMs. They are strictly pattern matchers with a huge dataset of patterns. There is also a problem with equating AI neural nodes and neurons in animal (including human) brains. Neurons do a lot more than fire when a threshold excitation occurs. There is stateful activity in the axons of some types of neurons (and neural nets have no diversity of types as neurons and other nervous system cells do). Also, neurons are awash in a bath of neurotransmitters, which varies in content over the geography of the brain and time. This acts as a set of regional and global variables which has no analog in neural nets. I could go on, but I’ve done this rant before in great detail and I don’t want to do it again. Just as a concluding remark: there is no evidence the brain computes in the sense of a Turing Machine and a lot of evidence that it does not.
Agreed, and introspection is inappropriate for studying the mind. Your subconscious doesn't exist to explain itself. It evolved for other purposes. Expect self serving stories from it when you introspect.
It will be interesting if this is the pin that pops the AI valuation bubble. It should undermine the demand for ever greater hyper-scaling. If so, it will deflate the big tech companies that are narrowly pushing up the stock market, not to mention Nvidia's huge valuation. It should also deflate Tesla, a hugely overvalued company whose value now seems to rest on self-driving cars and Musk's claims he will make household robot butlers.
It may well also pop the entire US economy, whose claims to growth rest largely on this mirage. Which is why the politicians are desperately colluding in the hype.
I don't believe the GOP politicians in control are intelligent and educated enough to understand how to manage the economy. If they did, they wouldn't be supporting Trump's economically disastrous actions. The supposedly competent ones, like Bessant, are just making up nonsense stories to support the unsupportable.
Wow. Reading your linked article from 2022 has totally blown my mind. How have the last 3 years gone the way they have when all your information was public and readily available?!
To give you some context about my own perspective:
• I’m 27 years old. I was born in 1998, which appears to be the year you started in this current research direction.
• In March 2022, when you published “Deep Learning Is Hitting a Wall”, I was three months away from graduating pharmacy school. I was focused on passing my board exams. Artificial intelligence wasn’t on my radar yet.
• My AI journey began when I watched a video about the 2023 “Sparks of AGI” paper.
• I made my Substack this year to ensure that we are incorporating patient-first principles and AI governance into healthcare.
I’m no computer scientist but I had come up with the need for neurosymbolic AI on my own. And the fact that people were talking about this in 1943 (!) makes me feel furious.
Like Cassandra in Greek mythology, you and many others laid out the problems ahead. Sadly, you were dismissed.
I don’t blame you if someone of your prestige and workload doesn’t care what a random Gen Z pharmacist has to say online. But my heart does break for you.
I hope you get whatever vindication you can out of this moment, and people start taking your ideas more seriously. Bravo and well-written.
Well, if you want to go further back, in 1990 I came to the US on a Fulbright Fellowship, just at the dawn of the digital revolution. When somebody introduced me to the World Wide Web - still just in academia - a lightbulb went off. In 1992 I made, with the BBC and PBS, 'The Electronic Frontier', which foresaw ubiquitous surveillance through smart devices, the death of copyright and Main Street, the computer in your pocket - 13 years before the iPhone - and Deepfakes, including their political risks. Before that, I'd made, also for the BBC, 'Robots Taking the Biscuit', which was a detailed examination of what it would take for a robot to bring you a cookie with your coffee. It's not just about calculation: imagine three small round yellow things being presented to a humanoid robot. One is a tennis ball, one a muffin and one a day old chick. It might 'recognise' them, it might have incredibly efficient haptic aand other sensors to pick them up, but in order to know what to do with each one it would need an entire sensorium plus experience of how they behave in the world. And in early 2020 I wrote a piece on Medium, 'Truth Decay', about the consequences of the dissolution of the world into digits. So, some of us have been banging on about this for many decades. Why would we even try to replace the extraordinary, unique thing that is our embodied intelligence, which is not only - as I said before - perfectly adapted to coexist with the rest of the living and physical world, but also able to do so on 24 watts of totally renewable energy? Beats me. More at www.sheilahayman.com
The global conversation is more nuanced than the headlines would have you believe. I would hazard that most people in the field have views that are a lot closer to Marcus than Altman.
I have a lowbrow answer: chatbots are fun and cool and they've got the "wow" factor. These companies are trying to make money and they figure attaching a chatbot to every last thing we interact with might result in some applications people will pay for.
Interesting question and I am sure other people will have different perspectives. For me I would say: First, the improvement in performance of these LLMs from, say, GPT-2 onwards, is immense; they just left our previous approaches in the dust. Second, it was discovered that scale mattered. Just making the same model bigger and training it with more data led to better and better results.
This has rewarded a greater focus on scale-up, engineering, and financial capital than innovation, invention and experimentation, although there has been plenty of that too (h/t "c-o-t reasoning" and innovations by DeepSeek and others).
The pendulum may swing back. You may not be aware that GPT architecture is actually a SIMPLIFICATION of approaches that were being tried out in the labs a decade ago. It was much more amenable to being broken down for parallel processing, however, and the rest is history. Current models learn abominably slowly. Once we get back to the lab, I think we will do better.
I would also point to the general conflation between linguistic competence and human intelligence. As Turing captured in his test, we all tend to associate human intelligence strongly with linguistic competence. The two actually decouple in interesting ways.
Finally, do keep an eye on DeepMind. They have always had a broader approach and have avoided drinking just the LLM Kool-Aid.
I think what DeepMind understand is that this technology is fantastically useful for specific, bounded purposes where clean and targeted data sets can be made. (This is why they were so keen to get hold of all our NHS data a few years ago). I don't think anyone would disagree with that.
Thanks so much for the insightful comment. It’s true that scale led to huge performance gains.
But whether it was misplaced hope or disingenuous marketing, the message was we were getting to AGI/ASI with the current paradigm in the next 5 years.
While GPT-5 certainly raised the ceiling of capabilities, I don’t know how much it meaningfully raised the floor. And the floor is what, in my opinion, will drive mass adoption in high-reliability industries.
Agree regarding DeepMind. I remember being blown away watching the AlphaGo documentary. Definitely following them closely to see what they do next.
Thanks again for all the insights. I appreciate it. Cheers!
AI will not save us. AI will not save big tech. They are zombie companies. My question is what to do with the over $600 billion incinerated in data center shaped trash cans, the most massive miss allocation of resources the world has ever known. All with the intention of making a few dozen men in Silicon Valley God emperors of the world. I don’t know how consolidating total power over our lives into the hands of these men is supposed to make things better. Any student with even the slightest overview of history, understands the problem with this.
Is the 2025 AI bubble gonna be as bad as the 2007 housing bubble? Maybe... The entire stock market depends on all these Mag 7 companies and their enormous AI CapEx budgets. As goes the Mag 7, so goes the S&P 500.
Generally, I think AI is about as bad as dot com and GFC combined. That's just a sense of things.
Investment in data centers is propping up GDP. At the very least, they'll be a shake out of the major players because there isn't enough economy to support all contenders.
The main reason why big tech companies have gone all in and even triple down on AI is because their PE is predicated on them remaining "growth" stocks. For years, they have been stringing out investors with promises of VR being the next big thing or whatever and year after year the next big thing doesn't materialize. Elon Musk is the undisputed king of this con.
Investors and the sector have a motivation to maintain the façade.
🎯 Prepare and invest accordingly. I'm bumping up my "crash insurance" funds. In 6 months when everyone knows that LLMs "AI" has hit a scaling wall and all this exorbitant investment is not gonna pay off, investors will react.
I think it’s always advisable to have 3 to 6 months of enough cash on hand to cover expenses. That’s typically basic financial advice. But I’ve also done things like have a pantry supplied with 5 kg bags of staples like rice and beans and a few other things just in case there are more direct practical necessities that suddenly become unavailable.
Corona really did change people’s outlook and they should not forget it
From memory, Rogue One was the prequel to the first Star Wars (an interesting sentence in itself) -- that movie concludes with the Death Star plans being handed to an AI-generated Princess Leia, presumably with six fingers.
Though (AI-generated Leia aside) to many - including myself -, Rogue One is actually one of the better / darker more-gritty "adult" Star Wars movies. Now that I'm no longer a child, the kitschy nature and rote plot lines of most Star Wars movies are generally pretty cringe.
Agree with the first Mr. S, take some exception to the second: the original Star Wars was arguably the greatest achievement in film in terms of overcoming obstacles and creating an incredibly creative new landscape (as for some of the writing & acting..."No comment, Senator"). Treat yourself to a copy of "The Making of Star Wars" by Rinzler; as Richard Harris says in 'Unforgiven': "You would stand, how shall I put it? In awe."
Did Gary *SEE* Rogue One? They STEAL the Death Star plans in that film. They blow up the Death Star in Star Wars, you know that great, classic film from 1977 that we've *all seen* dozens of times. 🙄
AGI is not possible with the transformer model LLM’s. You will never get rid of the hallucination problem and transformer models are unable to perform self-sanity checks precisely because they are transformer models.
The potential answer to AGI is in fact something wholly different than the transformer LLM architecture. We have a circle (the transformer model) and we need a square (AGI). The issues facing current LLM’s are intrinsic structural consequences and not mere issues of fine tuning.
CEOs for Claude and ChatGPT have “warned” the public of the havoc their systems would wreak on society solely as a way of weaponizing doom to further legitimize those systems. “If ChatGPT can create 20% unemployment, then surely it can help me with my thesis statement!”
Notice I don’t use “AI” to describe these platforms. They are transformer models, large language models, and nothing else. These companies will not use those terms because it inconveniently cuts the hype.
They can always have greater training data, but they will never achieve true introspection or intuition.
Haha, yes. I'm happy to see articles like the one Gary links to at the end, but c'mon, it's not like we haven't known from the start that LLMs are fragile and go from being really impressive to falling to pieces when pushed even slightly beyond the scope of their training. Pattern detection machines aren't gonna magically start performing deductive reasoning.
I asked chatGPT years ago, who said the famous quote "The industrial revolution and its consequences..." I figured that it could be reasonably expected to get that right, by "crowdsourcing" all the references to it. It told me Aldous Huxley said it.
Then I bullied it for a while, and every time, it would apologize and reverse itself. It was the ability to freely contradict itself that bothered me the most. I haven't "used" it since.
The biggest difference between Sam Altman and Elizabeth Holmes is that the Theranos fraud produced nearly immediate medical harm and investor outrage, where the harms of chatGPT are more diffuse and realized mostly by people foolish enough to take its outputs at face value (like the idiot advised to substitute sodium bromide for table salt.) But the AGI investor outrage singularity (AGIOS) approaches...
Seems to me Sam Altman is a full-on sociopath and anyone who believes what comes out of his mouth or invests in OpenAI is a sucker.
Thanks for this post. I read Karen Hao's excellent book, which is how I found you, Gary, because you made the most sense to me. Am heartened to see my instincts—and you—have been vindicated.
More people should be reading Karen Hao’s book, Gary’s Substack, and Ed Zitron on the insane financials. The people who trust this technology don’t seem to understand it at all.
Lambs to the slaughter. Amen.
Sam Altman is no different than other Tech Bros. Once either of them jumped on a hype wave and got a driving seat, it's like a one-way street. The only choice is to double down, build the hype even more, to suck even more VC dollars.
It's a separate thread, but what's absolutely fascinating to me is how uncritically anything AI is funded by investors.
* OpenAI's $300B valuation as of March 2025 (how far from potential profitability?)
* Thinking Machines Lab's $12B valuation before they even publicly revealed what they're working on
* Thousands of completely unknown startups that got 6-digit funding purely based on the premise of being AI startups
And that all before we fundamentally resolved what's allowed use of training data (from a law perspective).
One would think there would be a reflection that it doesn't look sustainable. But I guess I expect too much from VCs. Or underestimate the power of the combination of herd behaviors and greed.
And far, far too much money. People who have money to waste tend to spend it carelessly, particularly if their primary motive is greed for more money and power. I saw an interview of Thiel where he was asked which AI companies he's investing in; he responded to the effect he was investing in all the major players because he didn't know who would emerge on top. Not many have that kind of money to throw away. Also noticeable that people like Zuckerberg have ceased innovating and instead buy up all the competition and new innovative startups (i.e., Instagram and Whatsapp). They become lazy, greedy, monopolistic, and insulated from reality.
Yes, Altman is only one of them (i.e., Alex Karp, David Sacks, Elon Musk, Mark Andreesen, etc.), but it's not the only choice. My heroes are the ones who are doing the *work* needed to improve the human condition, increase prosperity for all, and take care of our beautiful blue planet. I admire Gary and Karen Hao for pursuing verifiable solutions to those ends in spite of ridicule and/or resistance from deluded trend followers. Let's hear it for a little humility, generosity, and collaboration:
"OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.
"Fostering its further development may be among the best things that companies, researchers, and governments can do. Investors, take note."
"Fostering its further development may be among the best things that companies, researchers, and governments can do. Investors, take note."
Now, believing that would assume that investors do listen to Gary Marcus and trust his judgment rather than default to herd behaviors.
That seems to me like a tall request.
But, admittedly, extensively working with and helping early-stage startups left me rather sour about VCs and their influence on the whole startup ecosystem. So, you can officially call me biased.
And those few heroes who go against the tide will remain unsung. The scale of their impact is simply too insignificant (when compared to hundreds of billions of dollars poured into AI darlings of the day).
Well, at least they were warned by one of their betters and Gary keeps it real for those who prefer to think for themselves.
Have you read "The Value of Everything: Making and Taking in the Global Economy" by Mariana Mazzucato?
"A scathing indictment of our current global financial system, The Value of Everything rigorously scrutinizes the way in which economic value has been accounted and reveals how economic theory has failed to clearly delineate the difference between value creation and value extraction. Mariana Mazzucato argues that the increasingly blurry distinction between the two categories has allowed certain actors in the economy to portray themselves as value creators, while in reality they are just moving around existing value or, even worse, destroying it.
"The book uses case studies—from Silicon Valley to the financial sector to big pharma—to show how the foggy notions of value create confusion between rents and profits, reward extractors and creators, and distort the measurements of growth and GDP. In the process, innovation suffers and inequality rises."
No, I haven't read it yet. I stumbled upon the title a couple of times, though. Thank you for the recommendation! It's on my list now.
"The unsinkable GP Titanic is preparing to depart. Last chance to make the voyage of a lifetime. All aboard!" -- Captain Altman, addressing VCs on the docks
It was sad, so sad, It was sad,
Sad when that Great ship went down
.. to the bottom of the ..
Despite all the alerts
The investors lost their shirts
It was sad when that great ship went down
Hey he’s gay give him a break
Gary gets it. 😎. OpenAI is a lab. They aren't a business. Subscriptions are nice but the real money comes in from VCs and SoftBank. chatGPT is not a product, it is a public experiment.
Before you bet your business on OpenAI, make sure you understand.
OpenAI hype is directly proportionate to capex, which is going nutz, and add in the — wait for it — "$500 billion valuation", and there you have it...Altman has to front-run GPT-5, as Anthropic doing more with less.
Maybe the AI bubble will burst before it gets really airborne.
or it may be something else altogther
I think we need more people amplifying Jonathan Shedler's observation "Al seems to know everything—until it's a topic where you have firsthand knowledge".
I think that's important to amplify, because I believe one of the biggest traps is when you're tempted to rely on LLM/GPT-based systems to answer questions on topics you don't understand or have little experience in. That adds significant risk to your ability to judge the quality and validity of the output.
It's something I've been saying in my critiques for more than a year now. I find that when I'm exploring ideas on a topic I know well, I more readily spot LLM/GPT-based system errors: then when attempting to adjust my prompts to correct for those, other errors - sometimes increasingly significant - can be produced. On topics I have some experience but lack both technical and practical depth, I still have sufficient empirical knowledge to spot problems: but when inquiring about domains I have minimal exposure to, how do I check for / confirm validity?
As a very experienced software developer, I feel comfortable using outputs of LLMs because I can verify that they work as expected and we can create concrete tests for evaluating code very easily. I keep a tight leash on generated code and test that it does what I expect. All of the code agents fail miserably at complex tasks and require human oversight.
To some extent, I share that long-term software-development experience. My early and then subsequent looks at LLM/GPT-based code-generation tools have been less-than-inspiring for a few reasons.
Possibly the two most-significant points were:
a) an apparent propensity to continuing the tradition of essentially "copy 'n paste" of commonly-used (and previously copied / shared code) that is available in large quantities in open-access code repositories. While that code can often be "good enough" as a starting point, it can propagate problems including poor (yet widely used) practices and outright errors: errors that although they may not surface *within* the context of an isolated single function or operation, can cause problems when that code is used as part of an integrated whole; and the follow-on problem,
b) an apparent inability of LLM/GPT-based tools to reason about a specific code fragment with regard to the integrated whole that code might reasonably be expected to exist within (a mixed-to-poor results when attempting to provide that specificity through prompts / starting context). By this I mean lack of consideration for temporal aspects (e.g. retention of state, multiple calls to the function), use of naming conventions consistently within the integrated system, and even more simply, lack of holistic consideration of error handling (e.g. over-compensating on some code sections or classes of error, and completely overlooking sections or classes). All of these considerations have a fundamental economic impact on code integration and maintenance.
That caused me to ponder a couple of questions:
How much of that "average" output is due to the nature of the code ingested to form the corpus? How much of the worlds truly exceptional, exemplar code is available to LLM's?
and that lead to thinking:
1) One might reasonably imagine the exceptional code is proprietary and hidden from view. If that's the case, do LLM's have access to the best code patterns that the worlds best developers / best systems use, or are generalised LLM corpora stuck with a slop of averages?
2) How much is the overarching context - the *why* or intent of a system - fundamentally intrinsic, important or instructive to the way in which the underlying code is best structured? One illustrative example here might be Domain-Specific Language's (DSL's). If that is a distinguishing character of better software-design, is AI-code generation using a generalised LLM a useful approach? Can generalised LLM/GPT-based tools understand the nuance of context - such as domain - and reliably and consistently use that nuance of specificity in filtering or forming the generated code? If not, does this require the development of specialised corpora for domain-specific, compliance or requirement-specific or even context-specific needs?
Perhaps a generalised LLM/GPT-based system can be useful (time or cost-saving) in brain-storming - exploring technical possibilities, or useful in comparing and exploring different technical options, or in domain-independent technical functions (e.g. compressing a file / image, encrypting / decrypting, etc), however, I don't yet see how the larger-context understanding needed to build good quality complete system is available / achievable with the commonly available, generalised LLM/GPT-based systems we have today.
Open Evidence is quite good when compared to a tool like UpToDate. I am not sure how or what OpenEvidence is trained on.
It's not a domain I'm a specialist in, so I can't really critique the validity specifically here.
However, I suspect that as a generalisation, it's likely that very specialised LLM/GPT-based systems that focus on a limited / targeted specialised domain with constrained inquiry, and a specifically-refined, carefully managed corpus curated by subject-experts will have better success than more generalised tools.
While this blog is great and Gary is great at criticizing the LLM approach, in the end of the day, in a very fundamental way, he actually is no different than Altman.
Basically he keeps on saying “funding my way of doing things is the only way to get to AGI.” Kids who believe in the tooth fairy have a more realistic view of the world. After all, they have hard evidence: experience shows them they get $1 every time they lose a tooth. No one and I mean no one on this planet has even the beginning of an idea of how to build a model that even partially explains human intelligence, consciousness and creativity. Not the slightest clue.
To pretend that AGI is even feasible without knowing anything about how the only existing AGI agents (human minds) operate, is just BS of the worst sort.
Machine “intelligence” is extremely useful in specific domains and should be researched and developed. But if we learn anything from the LLM hype machine it’s this: please STFU already about AGI.
Thank you. I have a huge admiration for Gary, and I just kinda mentally tuck away the occasional nods he gives to what needs to be done to achieve AGI. Hey, I don't need everyone to agree with me to value what they have to say.
But I'm entirely with you on this one. My prediction for the year AGI arrives is "never". Can't prove it, but I'd bet it against the field.
The only insight I agree with the AGI folks on is that intelligence didn't always exist on Earth but now it does and it's brought into being by some kind of physical (I'd prefer biophysical) system. So yeah, perhaps it's not *literaly impossible* for human beings to create machines that have "intelligence" similar to our own. But that's the most that can be said, anything beyond that is wild speculation.
And for perspective, us humans don't even know how to make life from non-life. That's a thing that happened somehow, right? There didn't used to be life on Earth, now there's life on Earth, must be some way to do it! But we don't know how. Seems if we're gonna prognosticate when future scientific discoveries will arrive, humans creating *any kind of living thing whatsoever* out of non-living matter ought to happen prior to humans figuring out how to create phenomena currently only known to exist in complex living things.
But hey, that's just me guessing. We're all just guessing.
| us humans don't even know how to make life from non-life.
| That's a thing that happened somehow, right? There didn't used to be
| life on Earth, now there's life on Earth, must be some way to do it!
That reminded me of the Miller-Urey experiment (1953), where basic inorganic compounds (methane, ammonia, hydrogen, and water) were subjected to electrical sparks (simulating lightning), heat and pressure enabling the formation of amino acids (organic basic building blocks of proteins). The experiment attempted in part to simulate what was believed to be conditions like that of early Earth, to test the hypothesis that organic molecules could form from inorganic compounds.
The subsequent 1961 Nirenberg and Matthaei experiment synthesised basic poly(U) RNA (and used that to synthesise a protein).
So perhaps it's fair to say that we have some established experiments that can create (synthesise) the basic molecular building blocks of life, and can reason about how those basics can occur in situ.
From there we can add some arguably hand-wavey-suppositions about how more complex microorganisms might mutate from those molecules in the presence of radiation - and / or could arrive on earth in frozen water from space rocks - to build a hypothesis of how life on earth.
edit: actually, the synthesis of RNA occurred earlier and is created to Severo Ochoa and collaborators in the 1950s. Nirenberg and Matthaei's work was important for using the synthetic RNA molecule (poly-U) to successfully direct the synthesis of a protein.
There are also alternative theories and experimental findings that suggest RNA could have been synthesized from simpler molecules in a prebiotic environment.
Thanks for the reply and the info. I probably overstated my case. My basic argument is that if we can't create artificial versions of very simple organisms, then there's little reason to speculate about us making an artificial version of a mysterious phenomenon that we only know to exist in organisms vastly more complex than the simple organisms that we don't know how to create artificially.
I can empathise with the frustration Aron.
I also agree it's important to research, ask about and critique the motivations of prominent public voices.
However, I'm not sure I can make the hops and jumps required to arrive at the claim of "funding my way of doing things is the only way to get to AGI".
In Gary, I see (as you note) a strong critique of the current myopic focus on a "hyper-space-race" based on a single pathway, using a single technique / approach, with obvious shortcomings and hard limits, that has largely devolved to a financial / commercial race to the bottom of finest hair-splitting minutia. The predominant focus appears to be on monetising the slop of a constrained (if not dead end) technology, instead of other possibilities such as investing in the alternative broad-research required to travel significantly further.
However, I don't see how Robust.AI (a company Gary has a key stake-holding / interest in) - with it's goal of creating an "off-the-shelf" machine-learning platform for autonomous robots - really qualifies as either AGI or a direct funding-competitor to the current LLM / GPT-based system "media darlings". I think that Robust.AI's focus arguably qualifies more as machine intelligence that "is extremely useful in specific domains and should be researched and developed".
My perception is that Gary is generally a) proposing less-specific / broader exploration: more openness to the exploration of alternatives (to LLM's), and b) a proponent of putting safeguards in place to potect humanity from the unintended and negative effects of AI systems.
Including these points in addition to style, I don't see the clear comparison / similarity with S Altman.
Regarding your claim that "no one on this planet has even the beginning of an idea of how to build a model that even partially explains human intelligence": you may ultimately be essentially correct, however that appears at first glance like an absolutist claim that would require significant knowledge to confirm.
I'd argue that statement as presented might be less true than you think.
If you haven't already, I'd encourage you to take a look at the OpenCog Hyperon work of Ben Goertzel and team, and the various projects utilising that ecosystem. My limited view is that Hyperion offers support for a broad and diverse range of approaches that collectively *may* begin to explore the diversity of reasoning processes humans have available.
OpenCog Hyperon is one project I know of, but I suspect there are likely others.
You would think. Also some of the data they need to train are behind paywalls.
I'm not an AI expert, nor A Reddit influencer, and I don't even use Twitter, so few people will listen. But for work I create mathematical models of real-world processes and I use Bayesian interference to tune their parameters. Not very different from what is done with LLMs. And what I posted more than a year ago somewhere on Substack was something like this:
The main difference between human (and animal!) intelligence on one side and current AI on the other is that we constantly create and adjust little models of reality that we use to make inferences. That's why our infants can learn new concepts on the basis of such a small sample size. But current AI only uses one humungous gigantic model with billions of parameters. It's no match for the creativity, flexibility and cognitive speed of real brains.
What I see now is that this is now increasingly being recognised as the problem of a lack of 'world models'. I am actually quite upbeat to see that it's going to be a long and winding path towards an automated ability to create world models. Time to really appreciate the depth and ingenuity of 'natural intelligence'!
I'd be a lot more upbeat if this 'long and winding road' wasn't destroying the planet and sucking up unfathomable amounts of money that is desperately needed elsewhere.
Valid comment. My hope is that as soon as investors realise it, they'll stop burning billions on it. Then Trump will surely step in, but even he tends to lose interest at some point.
Also likely though is that AGI will be given up and instead an artificial pseudo "intelligence" will be used to further build out a new system of slavery and surveillance empire backed up by automated weapons systems used against dissenters.
introspection is not appropriate for discovering how the human brain works.
Thank you for comparing this to human intelligence and phrasing it in a way I can understand it. The thought of various, adjusting models of reality pairs nicely with my general hypothesis that in order to have functioning intelligence a being needs to have senses!
I like and sgree with much of what you've said here Wolfgang.
My comments in response:
a) if you take a look at tge scope of what Wolfram Alpha does, I"m not certain the underlying models qualify as an "automated ability to create world models" and I would imagine that it's still relatively large (if not humongous/ gigantic). The focus here in those cases is on specialised domain-specific models that might be argued as somewhat bounded or finite, and thar need minimal tweaking once established.
b) I don't get the sense yet - at least not from public-facing comments - that the focus and hype around RNN systems of this specific LLM/GPT-based type is really b slowing or over. I think there's simply too much inflation and sunk investment for it to end without a significant "long tail" run out for the investment to be salvaged or used.
Your point about auto-generating little fallible models and tuning based on tests / feedback is excellent. Have you (or anyone else) seen any specific AI-based examples of this worth reviewing?
Yes, I think the focus will eventually be on creating more and more domain specific models. At some stage, the experience gained might then eventually lead to the ability to create ad hoc, situation specific models, as we ourselves do on a daily basis.
The current LLM focussed big-data model that tries to short circuit understanding will probably find some kind of justification for its existence, because so much money has been sunk into it. But the fact that only a handful of companies pursue this path is almost a guarantee for its eventual failure.
In nature, progress is the result of widely distributed experience and experimentation. Monocultures are not resilient have the tendency to collapse.
I'm a multi-award winning documentary filmmaker with a four-decade track record in films about technology and society. I have been trying for eight years, first via the MIT Media Lab and now via Cambridge University, to get people to understand that it's embodied intelligence via all our senses, and life in the world, that enables humans actually to understand things. Evolution over millions of years has also made us adaptive, versatile and extremely energy efficient - according to Stuart Russell, by a factor of about a million to one. Yann LeCun himself admits that a domestic cat has more actual intelligence than the most advanced 'AI' model. This hasn't changed in those eight years, yet the money thrown at this technology continues to balloon, with terrifying consequences for the climate and society. (And if it's any consolation, nobody is paying me for my work either).
The businesses making the investments either benefit directly from the hype (Amazon, Nvidia, OpenAI, Google Cloud, Amazon AWS, Microsoft Azure) or have huge existing franchises to defend that seem under threat from AI (Google Search, Microsoft Office, Meta, Apple). The politicians just hope that genAI can get them out of a productivity hole. I didn't expect the hype to last this long but that's a heck of a lot of momentum from those companies. And these tools are fun and remarkable even if they are nothing approaching intelligent.
Sorry, you lost me at "Guardian columnest'.
Nonsense, extremist newspaper, I'm afraid.
Exactly. Human intelligence is inseparable from a human body and human experience. Shouldn’t that be sort of obvious? Yet, it seems conveniently ignored or forgotten by some otherwise bright people in pursuit of some technological holy grail…
Yes, exactly. I have noticed many people think AI isn't intelligent because it can't do some behavior that humans do well but if we gave a machine all the senses and experiences humans actually get I hypothesize we might get something resembling intelligence.
Jeff Hawkins has proposed a different way to build an AGI and have open sourced their efforts here:
https://thousandbrains.org/
On a side note I think many readers would support microtransactions to support authors. I wrote the following to get the conversation started since the Substack team has made it evident that advertisements might be coming soon.
https://thelders.substack.com/p/should-substack-embrace-microtransactions
Excuse me Brad. Are you affirming that you believe that "if we gave a machine all the senses and experiences humans (get), that "we might get something resembling intelligence"?
Not affirming, hypothesizing.
Hey Sheila, DM me, I'd love to help connect you with philosophers, cognitive scientists, and research institutes that are sympathetic to this view. I certainly am.
Here's something I wrote a while back that can add some color on why embodiment is necessary for intelligence: https://tailwindthinking.substack.com/p/the-gnostic-cartesian-confusions
Blake, thanks a lot but I don't need introductions: as Director's Fellow at the MIT Media Lab and now as Advisory Board member of the Minderoo Centre at Cambridge University, over the past eight years I've interviewed/recruited dozens of top people in this area: a few who agree are Neil Lawrence, DeepMind Professor of Computer Science at Cambrudge, Christof Koch who runs AllenAI and Melanie Mitchell, computer science prof at the Santa Fe Institute. And, ironically in retrospect, when I first showed the first cut of the film at MIT, Mustafa Suleyman - then on the board of the Media Lab - told me it was 'really important' and I should 'get it out there as soon as possible'. I completely failed to recognise him so didn't take him up - I don't think he'd be quite so keen now, having thrown in his lot with Microsoft (I first interviewed Bill Gates, incidentally, in 1991 or so when he was still a geek wearing acrylic sweaters and bigging up Encarta, the CD Rom precursor of 'all the world's knowledge). So I don't need introductions! I need money to finish the thing and get it out into the world.
Would crowd-funding be viable?
Gotchya — maybe try reaching out to the folks at the Cosmos Institute? They would probably be interested in this kind of project. It could elevate their brand.
Thanks - I will take a look. Alison Gopnik is great. Not so inspired by the men you list; it's interesting to me how (with the obvious exception of people responding on this thread and a few others) very gendered this subject is. All the gung ho techno-positivist-entrepreneurs are men, and many of the most eloquent, erudite and expert sceptics are women; apart from Alison, Timnit Gebru, Meg Mitchell, Emily Bender, Melanie Mitchell, Margaret Wertheim and many others. I remember when I first presented my work at the MIT Media Lab, one of my co-Director's Fellows said to me, with pity in his voice, 'The problem you'll have, Sheila, is that these ideas aren't sexy'. By which I think he meant sexy as in 'send a giant phallus to Mars' Sexy, as opposed to 'engage in the subtle, laborious and complex work of sorting out our own planet' Unsexy. I guess developments in the eight years since have only borne him out.
Ha! Yes — skepticism about ambitigious technological projects is generally not sexy. But the story can easily be made sexy with a couple reframings. E.g. "elites misleading the public to enrich themselves" "techno-feudalist overlords desperately trying to satisfy their cosmic worldview" "triumph of gnosticism on the grandest scale ever seen" "tragic resurgence of man's hubristic attempt to surmount his humanity by mastering it" "technological stagnation and the last gasps of optimism"
The story about the quest for artificial intelligence is much older than the last few years, and the psychological motivations go back millennia. That's the story I would tell — of hubris, desperation for a theory of mind, desire to understand what makes humans different.
The story has to be "here again, we run up against the fundamental confusions in the project of trying to predict, explain, and replicate the complexity of biological life."
I hear you on the gender piece, but I wouldn't get too hung up on it. Framing optimism/pessimism as a gendered thing could hurt your story by needlessly alienating key audiences and making it seem like you have a political bone to pick. Also, don't sleep on Juliet Floyd! Just ask the female philosophers which men they'd recommend. They generally care far less about the gender stuff than about the ideas.
I wasn't saying I had any intention of publicly framing it like that. As to the ancient dream of living for ever/making artificial life/technology as religion, Sigal Samuel is fantastic on that https://www.vox.com/the-highlight/23779413/silicon-valleys-ai-religion-transhumanism-longtermism-ea?mc_cid=c83692ece6&mc_eid=3bec0f0e90
(as, indeed, is Margaret Wertheim's 'Pythagoras' Trousers'.) My film/films are aimed at wide general audiences, so I prefer to frame the argument in cats, cookies, spiders, toddlers, cab drivers, nurses, truckdrivers etc. More jokes in those areas and more familiar 'aha!' moments. But thanks.
Also for philosophers, I’d recommend Alva Noë, Anton Ford, Andy Clark, and Juliet Floyd.
Cognitive scientists: Allison Gopnik, Dietrich Stout, Rolf Pfeifer, Pieter Abbeel, Josh Tenenbaum.
The thing that I find most troubling about the launch was just how desperately people cried for a return to the sycophantic style of 4o — because it was more engaging for the companion/therapist/life coach uses that have come to characterize most people's relationships with GenAI. Sam acknowledged that this was sad and gross, but caved anyway and gave them 4o.
This would have all been inconceivable two years ago. I'm really curious how the street will react on Monday.
Thanks for your writing on this. You've earned a victory lap — and a break.
The sycophantic nature of 4o and the Claude models kind of scares me a bit. Really quick to tell you how amazing and brilliant you are.
It's how they juice their user/subscriber numbers to try and convince investors that the money is coming. It's extremely gross and sad and exploitative. Watching people have full on mental breakdowns because o4 got shut down made me feel dirty and quite angry that it was allowed to happen.
I expect a lot of people inside OpenAI feel the same way. There has to be growing disillusionment. They thought they’d unlock human potential and it turns out they’re just burying us deeper in the solipsism of our screens.
I can make 4o be less sycophantic, but I can't seem to make 5 be funny.
In 1957-59 I studied numerical methods under T. S. Motzkin; I was an undergraduate but at his invitation was able to participate in his graduate seminar. I was also very interested in the foundations of mathematics and formal logic. This was in the immediate aftermath of the Dartmouth Workshop and everyone I knew was very excited about "artificial intelligence." But based on my own knowledge of mathematics and computers and after some discussions with Motzkin and others I concluded that AI was going to remain a very limited model of what is generally understood by the word "intelligence" unless or until some very profound and unforeseen discoveries were to come about. As wave after wave of enthusiasm for AI has risen in the intervening 78 years I have scanned each for evidence of fundamental transformation but never finding any. Nothing has ever been done that could not be analyzed in the terms known to T.S. Motzin and other first-class mathematicians of his era. We long ago reached the limits of what is possible by engineering tweaks to AI programs. Anyone who truly wishes to pursue AGI, it seems to me, must seek to better understand just how the brain creates the mind.
This is a fundamental arrogance of people in the AI space. We don't fully understand the brain or the mind yet but have the hubris to believe we can create the equivalent. Having a paper confirm that it is just an imitation and not the real thing is nice but it's always just been an imitation. These tools try to mimic the output that comes from humans but do nothing to emulate how a brain actually works despite using terms like "neural networks."
I think this hits the nail on the head. The common thinking is that AGI = human intelligence. Ask any group of Neuroscientist where we are in the understanding of human intelligence you’ll see a lot of shrugged shoulders. Without a good model of the human brain/mind, the best thing you can do is the equivalent of throwing paint against a wall trying to get an exact replica of the Mona Lisa with nothing more than a stick figure drawing for reference.
There have been several state sponsored programs going on in various counties for the last 10 to 20 years. The US has one, China has one. The EU shut down the Human Brain project in 2023. They learned a lot, but any goal of a complete understanding of the human brain was not achieved.
I think we have a long way to go before we have a deep enough understanding of the brain/mind to create human equivalent AGI.
Will, bingo. Specifically the culprit is the PSSH - the questionable hypothesis on which ALL AI rests!!
Indeed, the PSS hypothesis is certainly a very major and fundamental part of the problem.
My understanding was that PSSH was narrower and would not include neural networks, or their digital simulation, in LLMs?
As formulated originally by Newell and Simon, their physical symbol system hypothesis (PSSH) is altogether independent of the mechanics of implementation. As they say in their landmark 1976 ACM paper:
"The Physical Symbol System Hypothesis. A physical symbol system has the necessary and sufficient means for general intelligent action." Wikipedia has a useful brief summary.
Thanks. The Wikipedia article suggests that PSSH, at least in its stronger form, excludes neural networks. I don’t have a dog in this fight, so happy to include neural networks under the umbrella if that makes sense. I certainly think about LLMs symbolically and am confident they can be understood this way.
I don’t think you can attribute any sort of symbolic functionality to LLMs. They are strictly pattern matchers with a huge dataset of patterns. There is also a problem with equating AI neural nodes and neurons in animal (including human) brains. Neurons do a lot more than fire when a threshold excitation occurs. There is stateful activity in the axons of some types of neurons (and neural nets have no diversity of types as neurons and other nervous system cells do). Also, neurons are awash in a bath of neurotransmitters, which varies in content over the geography of the brain and time. This acts as a set of regional and global variables which has no analog in neural nets. I could go on, but I’ve done this rant before in great detail and I don’t want to do it again. Just as a concluding remark: there is no evidence the brain computes in the sense of a Turing Machine and a lot of evidence that it does not.
Agreed, and introspection is inappropriate for studying the mind. Your subconscious doesn't exist to explain itself. It evolved for other purposes. Expect self serving stories from it when you introspect.
It will be interesting if this is the pin that pops the AI valuation bubble. It should undermine the demand for ever greater hyper-scaling. If so, it will deflate the big tech companies that are narrowly pushing up the stock market, not to mention Nvidia's huge valuation. It should also deflate Tesla, a hugely overvalued company whose value now seems to rest on self-driving cars and Musk's claims he will make household robot butlers.
It may well also pop the entire US economy, whose claims to growth rest largely on this mirage. Which is why the politicians are desperately colluding in the hype.
I don't believe the GOP politicians in control are intelligent and educated enough to understand how to manage the economy. If they did, they wouldn't be supporting Trump's economically disastrous actions. The supposedly competent ones, like Bessant, are just making up nonsense stories to support the unsupportable.
Yes, that auto switching mechanism is never going to work well. Because AI has no self-reflection. It doesn't know what it doesn't know.
Therefore, it cannot properly delegate to the correct model that does know.
Wow. Reading your linked article from 2022 has totally blown my mind. How have the last 3 years gone the way they have when all your information was public and readily available?!
To give you some context about my own perspective:
• I’m 27 years old. I was born in 1998, which appears to be the year you started in this current research direction.
• In March 2022, when you published “Deep Learning Is Hitting a Wall”, I was three months away from graduating pharmacy school. I was focused on passing my board exams. Artificial intelligence wasn’t on my radar yet.
• My AI journey began when I watched a video about the 2023 “Sparks of AGI” paper.
• I made my Substack this year to ensure that we are incorporating patient-first principles and AI governance into healthcare.
I’m no computer scientist but I had come up with the need for neurosymbolic AI on my own. And the fact that people were talking about this in 1943 (!) makes me feel furious.
Like Cassandra in Greek mythology, you and many others laid out the problems ahead. Sadly, you were dismissed.
I don’t blame you if someone of your prestige and workload doesn’t care what a random Gen Z pharmacist has to say online. But my heart does break for you.
I hope you get whatever vindication you can out of this moment, and people start taking your ideas more seriously. Bravo and well-written.
Well, if you want to go further back, in 1990 I came to the US on a Fulbright Fellowship, just at the dawn of the digital revolution. When somebody introduced me to the World Wide Web - still just in academia - a lightbulb went off. In 1992 I made, with the BBC and PBS, 'The Electronic Frontier', which foresaw ubiquitous surveillance through smart devices, the death of copyright and Main Street, the computer in your pocket - 13 years before the iPhone - and Deepfakes, including their political risks. Before that, I'd made, also for the BBC, 'Robots Taking the Biscuit', which was a detailed examination of what it would take for a robot to bring you a cookie with your coffee. It's not just about calculation: imagine three small round yellow things being presented to a humanoid robot. One is a tennis ball, one a muffin and one a day old chick. It might 'recognise' them, it might have incredibly efficient haptic aand other sensors to pick them up, but in order to know what to do with each one it would need an entire sensorium plus experience of how they behave in the world. And in early 2020 I wrote a piece on Medium, 'Truth Decay', about the consequences of the dissolution of the world into digits. So, some of us have been banging on about this for many decades. Why would we even try to replace the extraordinary, unique thing that is our embodied intelligence, which is not only - as I said before - perfectly adapted to coexist with the rest of the living and physical world, but also able to do so on 24 watts of totally renewable energy? Beats me. More at www.sheilahayman.com
Talk about prescient. That’s amazing! Thanks for all your work and thoughtfulness.
The global conversation is more nuanced than the headlines would have you believe. I would hazard that most people in the field have views that are a lot closer to Marcus than Altman.
I can believe that, but why did all the leading labs go for a pure LLM-based approach? At least from what we can tell from the outside.
I have a lowbrow answer: chatbots are fun and cool and they've got the "wow" factor. These companies are trying to make money and they figure attaching a chatbot to every last thing we interact with might result in some applications people will pay for.
Simplicity is the ultimate sophistication. I am sure you’re right, and everything else is just marketing.
Interesting question and I am sure other people will have different perspectives. For me I would say: First, the improvement in performance of these LLMs from, say, GPT-2 onwards, is immense; they just left our previous approaches in the dust. Second, it was discovered that scale mattered. Just making the same model bigger and training it with more data led to better and better results.
This has rewarded a greater focus on scale-up, engineering, and financial capital than innovation, invention and experimentation, although there has been plenty of that too (h/t "c-o-t reasoning" and innovations by DeepSeek and others).
The pendulum may swing back. You may not be aware that GPT architecture is actually a SIMPLIFICATION of approaches that were being tried out in the labs a decade ago. It was much more amenable to being broken down for parallel processing, however, and the rest is history. Current models learn abominably slowly. Once we get back to the lab, I think we will do better.
I would also point to the general conflation between linguistic competence and human intelligence. As Turing captured in his test, we all tend to associate human intelligence strongly with linguistic competence. The two actually decouple in interesting ways.
Finally, do keep an eye on DeepMind. They have always had a broader approach and have avoided drinking just the LLM Kool-Aid.
I think what DeepMind understand is that this technology is fantastically useful for specific, bounded purposes where clean and targeted data sets can be made. (This is why they were so keen to get hold of all our NHS data a few years ago). I don't think anyone would disagree with that.
Thanks so much for the insightful comment. It’s true that scale led to huge performance gains.
But whether it was misplaced hope or disingenuous marketing, the message was we were getting to AGI/ASI with the current paradigm in the next 5 years.
While GPT-5 certainly raised the ceiling of capabilities, I don’t know how much it meaningfully raised the floor. And the floor is what, in my opinion, will drive mass adoption in high-reliability industries.
Agree regarding DeepMind. I remember being blown away watching the AlphaGo documentary. Definitely following them closely to see what they do next.
Thanks again for all the insights. I appreciate it. Cheers!
AI will not save us. AI will not save big tech. They are zombie companies. My question is what to do with the over $600 billion incinerated in data center shaped trash cans, the most massive miss allocation of resources the world has ever known. All with the intention of making a few dozen men in Silicon Valley God emperors of the world. I don’t know how consolidating total power over our lives into the hands of these men is supposed to make things better. Any student with even the slightest overview of history, understands the problem with this.
Is the 2025 AI bubble gonna be as bad as the 2007 housing bubble? Maybe... The entire stock market depends on all these Mag 7 companies and their enormous AI CapEx budgets. As goes the Mag 7, so goes the S&P 500.
3 small AI companies down today Monday 8/11:
Monday.com -30%
BigBear.ai -29%
C3.ai -26%
Shareholders lost a combined $5.2 Billion
All because of disappointing results and guidance
Is AI overhyped? 🤔
Generally, I think AI is about as bad as dot com and GFC combined. That's just a sense of things.
Investment in data centers is propping up GDP. At the very least, they'll be a shake out of the major players because there isn't enough economy to support all contenders.
The main reason why big tech companies have gone all in and even triple down on AI is because their PE is predicated on them remaining "growth" stocks. For years, they have been stringing out investors with promises of VR being the next big thing or whatever and year after year the next big thing doesn't materialize. Elon Musk is the undisputed king of this con.
Investors and the sector have a motivation to maintain the façade.
🎯 Prepare and invest accordingly. I'm bumping up my "crash insurance" funds. In 6 months when everyone knows that LLMs "AI" has hit a scaling wall and all this exorbitant investment is not gonna pay off, investors will react.
I think it’s always advisable to have 3 to 6 months of enough cash on hand to cover expenses. That’s typically basic financial advice. But I’ve also done things like have a pantry supplied with 5 kg bags of staples like rice and beans and a few other things just in case there are more direct practical necessities that suddenly become unavailable.
Corona really did change people’s outlook and they should not forget it
Mother of mercy, Gary -- the Rebel Alliance did NOT blow up the Death Star in Rogue One.
(Did you get that answer from GPT-5?)
can you help me fix it? i think i must have garbled something others told me. (i mostly got off the bus after the original 3)
From memory, Rogue One was the prequel to the first Star Wars (an interesting sentence in itself) -- that movie concludes with the Death Star plans being handed to an AI-generated Princess Leia, presumably with six fingers.
What is this "Star Wars" you speak of?
https://www.youtube.com/watch?v=Cg-pnGFbwMQ
Yes, decidedly downhill after the original two. (The third one could've gotten Rosa Parks off the bus.)
Though (AI-generated Leia aside) to many - including myself -, Rogue One is actually one of the better / darker more-gritty "adult" Star Wars movies. Now that I'm no longer a child, the kitschy nature and rote plot lines of most Star Wars movies are generally pretty cringe.
Rogue One is the best Star Wars movie, period.
Agree with the first Mr. S, take some exception to the second: the original Star Wars was arguably the greatest achievement in film in terms of overcoming obstacles and creating an incredibly creative new landscape (as for some of the writing & acting..."No comment, Senator"). Treat yourself to a copy of "The Making of Star Wars" by Rinzler; as Richard Harris says in 'Unforgiven': "You would stand, how shall I put it? In awe."
No fix needed, the Rebel Alliance does indeed blow up the Death Star in the next movie in the timeline, which is the original Star Wars.
Did Gary *SEE* Rogue One? They STEAL the Death Star plans in that film. They blow up the Death Star in Star Wars, you know that great, classic film from 1977 that we've *all seen* dozens of times. 🙄
We still love you, Gary.
AGI is not possible with the transformer model LLM’s. You will never get rid of the hallucination problem and transformer models are unable to perform self-sanity checks precisely because they are transformer models.
The potential answer to AGI is in fact something wholly different than the transformer LLM architecture. We have a circle (the transformer model) and we need a square (AGI). The issues facing current LLM’s are intrinsic structural consequences and not mere issues of fine tuning.
CEOs for Claude and ChatGPT have “warned” the public of the havoc their systems would wreak on society solely as a way of weaponizing doom to further legitimize those systems. “If ChatGPT can create 20% unemployment, then surely it can help me with my thesis statement!”
Notice I don’t use “AI” to describe these platforms. They are transformer models, large language models, and nothing else. These companies will not use those terms because it inconveniently cuts the hype.
They can always have greater training data, but they will never achieve true introspection or intuition.
Thank you for the good article.
I'm worried about the people who need good articles and scientific papers to realize that this sh*t sucks.
LLMs have taught us a lot about people, and what we've learned is not encouraging.
Haha, yes. I'm happy to see articles like the one Gary links to at the end, but c'mon, it's not like we haven't known from the start that LLMs are fragile and go from being really impressive to falling to pieces when pushed even slightly beyond the scope of their training. Pattern detection machines aren't gonna magically start performing deductive reasoning.
I don't really touch this stuff tbh.
I asked chatGPT years ago, who said the famous quote "The industrial revolution and its consequences..." I figured that it could be reasonably expected to get that right, by "crowdsourcing" all the references to it. It told me Aldous Huxley said it.
Then I bullied it for a while, and every time, it would apologize and reverse itself. It was the ability to freely contradict itself that bothered me the most. I haven't "used" it since.
Q: what about the third b?
A: Ah, you’re right—there is a third b in “blueberry”! Let’s look carefully again:
b l u e b e b r r y
There are actually 3 occurrences of the letter b.
Thanks for catching that!
Haha, I absolutely love watching what happens when you insinuate a false fact.
And a bluebebrry drink is a bluebeberrage, right?
The biggest difference between Sam Altman and Elizabeth Holmes is that the Theranos fraud produced nearly immediate medical harm and investor outrage, where the harms of chatGPT are more diffuse and realized mostly by people foolish enough to take its outputs at face value (like the idiot advised to substitute sodium bromide for table salt.) But the AGI investor outrage singularity (AGIOS) approaches...