Gary, I heard you list 10 great reasons for skepticism on the BBC. The interviewer acknowledged you with mild interest and then gave equal time to an investor who shared hype and zero counter evidence.
Meanwhile, the hype has every business searching for use cases, most of which are pretty meh. An AI Assistant is just a form, built with trial and error, to reduce trial and error for someone building a prompt. And the output still likely includes errors. And the Assistant works okay for a few months, and then stops working, and the people who created it don’t know why. What a massive waste of time.
Although I am also an LLM skeptic, I do want to point out that it is hard for investors and AI gurus to offer anything approaching evidence for their enthusiasm. They think, or pretend to think, that their technology will get better by future scaling and/or unspecified incremental algorithm improvement. It is very difficult to prove that they can't get there. Still, I have great confidence that they won't.
Interesting thought! Isn’t it generally harder to prove a negative? I heard a safety head from open AI insisting that AGI is coming very soon, and then mentioned that one of his concerns is that no one is paying any attention to solving hallucinations. The contradiction in those claims is amazing. Sure, all of his financial incentives are telling him to ignore it.
I have no opinion on the long term for AI. It will be here in some form, for good or ill, unless there is a massive backlash to the harms.
It’s easy to claim AGI is coming soon — and even that you have achieved it — when you haven’t ever precisely defined what you mean by AGI.
Or maybe AGI is like obscenity and we will all know it when we see it?
Given that one of the most prevalent uses of generative AI is bound to be porn production, the latter may actually be pretty close to the truth: when a porn conosewer can’t tell the difference between AI generated video and video of the real thing, we will know that AGI (Artificial Genital Intelligence) has arrived.
But first, I suspect the AI engineers will have to fix the inevitable random swapping of male and female body parts in their generated video .
It's not only a massive waste of time the intellectual rot it has spawned has reached scientific publishing.
In their paper the authors wrote:
"Our findings suggest that the N-terminus of SN1 may act as a nucleation site for SNARE acceptor complex assembly. As a side note, our result of an intrinsic disorder of SNAP25 contrasts with a recent prediction of AlphaFold2 that predicted SN1 and SN2 to be entirely α-helical with high confidence." [1]
Interestingly the last sentence was eliminated in the published paper. Elsevier sells a number of AI journals so any well-founded criticism threatens one of their profit streams. Even mild criticism of AI is, thus, not allowed.
[1] Stief T, Gremer L, Pribicevic S, Espinueva DF, Vormann K, Biehl R, Jahn R, Pérez-Lara Á, Lakomek NA. Intrinsic Disorder of the Neuronal SNARE Protein SNAP25a in its Pre-fusion Conformation. J Mol Biol. 2023 May 15;435(10):168069. doi: 10.1016/j.jmb.2023.168069. Epub 2023 Mar 30. PMID: 37003471.
Yeah, journalists call that "balance". They did the same thing with climate change denial. Give one side to people who have no clue what they are talking about, and the other side to people who do.
They have zero clue what actual objectivity, or balance is. Its just the truth, nothing more. Its not entirely journalists fault though, given the corruption of data scientists in the tech industry. They don't know what the science is. They do know that they should seek out all scientific opinion and get a good sense of it.
I listened to Gary Marcus on radio new zealand, and it was a fine interview. But they just had to put an AI doomer on nextweek to balance it out. It bothers me because that radio station is not even commercialized. Its the last bastion of media that isn't corporate or capitalist controlled, or runs for profit.
You have a good point but I started to discover young developers with a new approach such https://www.cursor.com/
I discovered them both by the Lex Fridman podcast and I discovered Elliott who’s 20 and has a similar approach. They’re all extremely young but it’s interesting to listen to them and it’s very easy to contact them. Moreover, I think Elon never tried to censor you: you clashed about politics and you attacked the Tesla Robotaxi event: let it go, it’s just marketing. Please come back in the Adrian space. Thanks.
Well written. Speaking as a student of technological change, this is not surprising. Nor is it surprising that legions of fans deny it!
All exponential improvement curves eventually turn into S curves (i.e. slowing rate of improvement) as one or more fundamental limits start to bear. They can last for a long time - look at multiple versions of Moore’s Law lasting 50+ years! But they cannot go on forever. (“If something cannot last forever, it won’t. I forget which SF author coined that.)
Fortunately for technology, another approach can eventually become relevant and its performance can surpass the old one. So the overall rate of improvement can look vaguely exponential, even though it’s a series of S curves. In fact LLMs in some ways were an example - a very different approach to AI rocketed past machine learning approaches, for some applications.
Of course this pattern is not proof, by itself, that LLMs are slowing down. But if not now, they will eventually. Others have written about why that will happen before “General AI” levels of performance.
However, as per OpenAIs own research, transformer based LLMs had already started to plateau in performance around gpt3.5. These models don’t reason, they memorize word patterns and act as extremely efficient Markov chain models. And there is no reason to believe that such a model , no matter how large the training data or parameter size will ever be able to reason. Still, as tools and simple question answer machines, they are very useful. They actually make us marvel at the awesomeness of brains and neurons, which can do so much with a fraction of the energy consumption of these constructs.
It’s not even a question of S curves though. Otherwise intelligent people literally believed that human intelligence was a 1950s-era neural net model merely accelerated with enough GPU cycle speeds and a massive data set.
Talk about riding reductionism off a Wile E. Coyote cliff.
May be off a slightly different tangent...I get the feeling it's a similar kind of operationalism that Watson, later Skinner introduced. The kind of operationalism that is also part of Steven's measurement model that dominates social sciences: as long as you map the empirical relations to numerical ones more or less consistently, you have measured. Measured what exactly? Doesn't matter, the model does not say, who cares? Turing's eponymous test is of the same kind, not even validated, just asserted. As long as current AI activities live in that bubble nothing will change. Even if the bubble bursts, nothing will change. "Let's not dwell on the past, let's move forward...". There won't be a truth commission. Current AI is good enough a pretext to fire people, hire younger ones at lower salaries in countries with lower labor regulations. Bottomline improves, profit improves, almost everyone is happy. And for all the expenses and unused graphic cards, there will be special tax regulations to ease the burden on the industry. After all, everybody was wrong, it was only business, not personal.
So, it sounds like another Theranos situation where the media never bother to really talk/listen to the pathologists (or, in this case the scientists/researchers in the AI trenches) and instead only listen to those with a vested interest.
I think the Theranos comparison is unfair. Theranos committed outright fraud. I don't hear the AI companies being accused of that. As far as I know, there's no law against unrealistic expectations or simply bad reasoning. Many companies fail because the stuff they're working on doesn't fulfill expectations. It's not fraud.
I largely agree with you. I mostly don't believe anyone is out and out perpetuating fraud. Still, when I hear Sam Altman or (in the past) Mira Murati claim in interviews that they're seeing signs of "intelligence" " emerging" from their next-gen models, I have to wonder. Could they possibly believe this? Obviously they have a lot of financial incentive to drink their own Kool-Aid. Do they believe it in the abstract and feel like they need to buy more time to make it happen? I don't know how to interpret it.
Who knows what they truly believe? It doesn't really matter because they are free to hold whatever opinions they want. It is usual for CEOs and CTOs to hype their products. It is on us to decide whether what they say is useful, with the help of experts like Gary Marcus of course.
Until the algorithms change radically from the LLM technology they are all using now, I would dismiss all comments about "intelligence emerging". Whatever intelligence they output is derived from the human intelligence embodied in their training data, which is pretty much the entire internet. They are improving their products by scaling (ie, more training data) and little tweaks to try to control their output (eg, adhere to social norms, filter in favor of truth). As far as I can tell, they have no plan to reach AGI other than a general desire to do so.
You are right, of course. However, I could care less whether its fraud or people choosing to believe whatever they want.
If people who have all the money in the world and every opportunity to listen to criticism want to lie to themselves, and it causes a lot of harm in the process, then its just a clever way of gaming the system. Fraud at least allows them to be put in jail more easily.
I'd prefer to just follow my moral intuitions here than accept some arbitrary law.
Aside from that, LLMs/AI is a much bigger industry than Theranos, which was small potatoes compared to this really.
Fair - I think it wasn't Theranos - but might be about to tip over into that. If Open AI et al. have discovered what they are selling (GAI) is bunk, then they will be commiting fraud when they go to the market and sell it as such.
Certainly there is motivation to do fraud when there's so much money riding on it. On the other hand, the nature of AI and LLMs is that a company has to release it into the wild before people will regard it as real, regardless of their talk. That makes it hard to commit fraud.
Unrealistic expectations in and of itself are not fraud. However, selling those unrealistic expectations while you know they can't be met in any way, is. That is precisely what Theranos was doing, selling something they knew they couldn't actually deliver. I fail to see the difference in what OpenAI is selling to the world. And if the OpenAI crowd isn't aware of this, then they are merely delusional.
I believe Theranos’ fraud was telling people their blood was being processed by their invention while actually sending it to a regular lab. They stepped over the line. As far as I know, OpenAI hasn’t done anything like this. What they are actually selling can be tried out by their customers. Telling people that AGI is just around the corner is just hype and legal. Of course, if it is seen as unrealistic, it will damage their reputation and investors will bail. I think we are at that point now.
Actually, she was not convicted for that specifically, she was convicted for telling investors she could do something that she knowingly couldn't. The big debate, after the whole Theranos debacle, was about the Silicon Valley idea of "fake it until you make it". OpenAI is also telling investors they can do something (in the near future) that they know they cannot. The only difference with Theranos is that the actual risk for people using it, is harder to understand (fake blood-testing results vs fake information).
Regardless of whether a particular case rises to the level of fraud in the legal sense, one thing is clear: the “fake it till you make it” crowd give legitimate scientists and engineers a bad name.
I knew LeCun initially panned you and then slyly realigned himself closer to you (I remember asking on an earlier article of yours if he'd come out and admitted as much) but this is the first I'm hearing of him trying to get you deplatformed. What an odious creature.
LeCun never responded to that specific essay, but yes he did call me a grifter, and I blocked him for the ad hominem attack. he blocked me back. i have offered a truce; he has refused to respond.
If you rewrote this article _without_ the _eighteen_ uses of I, me, and my, people might take it more seriously. Otherwise some might think that this is entirely an ego exercise on your part.
I think you are asking too much! Find one person who takes this much beating from grifters and swindlers and are still standing and churning out post after post. I am sure for many individuals in the industry Gary's non-existence is preferrable to his existence. It's easy for you to preach for calm and balanced tone when you are not the one in the ring fighting.
When the attacks go to one's character/integrity/intelligence/quality of work...that is inherently personal. I feel for you...as I think that anyone paying attention must conclude that you have been a public beacon of intellectual honesty and integrity...and that others/biz have not...for obvious reasons ($$/power). And using 'it's only business, nothing personal'...often said with a wink...is still a cruel joke for those of us that try to remain human.
See that is interesting. When I read this Gary's article first time I did not even notice those "I, me, my" etc. I simply got a timeline of Gary's predictions and subsequent findings. After I read your post and went back to read the same article again, those "I, me, my" still did not catch my attention. See I pay attention to what were said, not how they were said, because they are irrelevant.
If what he stated were facts, what does it matter if it appears self-promoting or not to someone? In that case it is that someone's problem not his. If what he stated were not facts, then point out those non-facts please! Don't just put out "opinion" without evidence.
Inevitable plateau. We could say the same about large image classifier and object detector models based on monolithic 'lets just add more data in even bigger networks' CNNs and R-CNNs. But, as more hybridisation takes place (e.g. adding logic and automated reasoning branches of AI, multi agent solutions, etc.) their capabilities will rise again. No longer LLMs then? Cheating? Who cares.
LLMs have been pushed into applications which go beyond their language capabilities. Inevitable. But hybridisation and or integration and systems engineering methods can come to the rescue to make them work well enough for more albeit not all applications.
Thanks for being a voice of reason. My guess is that we reached a stagnation phase similar to the one we saw with a lot of other technologies that were thought to be on the verge of a major breakthrough before things slowed down.
Transformer-based LLMs are definitely something to remember as we move forward with AI R&D, but they should not be a stepping stone; this is my opinion and what I am practicing.
Learning about Transformers gave me an idea to algorithmically correlate words to each other.
The simplest useful relationship can be expressed as a trigram: previous word, current word, and next word.
Using those trigrams, a Natural Language Generator (NLG) that I wrote makes grammatically-correct sentences not in the training set such as the ones below:
====== NLG-generated ==========
I want to speak more.
I want to change her tune.
================
There is too much overhead and uncertainty in GenAI's Transformers to make them a foundation for future AI R&D.
This article has _lowered_ my estimate of probability of LLMs hitting the wall.
First, there's very little in terms of substance and very much in terms of "I told you so" and accusing others of being alchemists. (What's next? LLM as phlogiston?)
Second, it quotes Marc Andreesen, which is a sure source for "listen and do the reverse".
The first link? The 2022 opinion piece on how we need to solve hallucination and abstraction to reason?
I'll be generous and presume, deducing it from your other comment, that you meant the _second_ link. It is indeed an argument for the trend being less exponential that some expect. (Not an utterly unbeatable argument, mind you, but an argument nonetheless) But if the growth becomes, say, "merely" quadratic, it will still deliver quite some results for years to come.
the initial evidence was linked at top near the word April. a lot of recent evidence is proprietary, leaking out, but clearly consistent with what i anticipated.
I appreciate your perspective in these times of AI hype. What companies or research groups do you think are making the most interesting progress in synthesizing logic/reasoning with reference data to excel where LLMs fall short? Do the most promising approaches even involve neural nets?
I can't think of a time when public figures have been so publicly confident of something that's pure faith. It's more akin to religion than science! And yet they will shout you down so fast if you point to the naked emperor.
Gary, I heard you list 10 great reasons for skepticism on the BBC. The interviewer acknowledged you with mild interest and then gave equal time to an investor who shared hype and zero counter evidence.
Meanwhile, the hype has every business searching for use cases, most of which are pretty meh. An AI Assistant is just a form, built with trial and error, to reduce trial and error for someone building a prompt. And the output still likely includes errors. And the Assistant works okay for a few months, and then stops working, and the people who created it don’t know why. What a massive waste of time.
your first paragraph is so damning, and a tremendous illustration of what went wrong. i will quote it sometime!
Although I am also an LLM skeptic, I do want to point out that it is hard for investors and AI gurus to offer anything approaching evidence for their enthusiasm. They think, or pretend to think, that their technology will get better by future scaling and/or unspecified incremental algorithm improvement. It is very difficult to prove that they can't get there. Still, I have great confidence that they won't.
Interesting thought! Isn’t it generally harder to prove a negative? I heard a safety head from open AI insisting that AGI is coming very soon, and then mentioned that one of his concerns is that no one is paying any attention to solving hallucinations. The contradiction in those claims is amazing. Sure, all of his financial incentives are telling him to ignore it.
I have no opinion on the long term for AI. It will be here in some form, for good or ill, unless there is a massive backlash to the harms.
It’s easy to claim AGI is coming soon — and even that you have achieved it — when you haven’t ever precisely defined what you mean by AGI.
Or maybe AGI is like obscenity and we will all know it when we see it?
Given that one of the most prevalent uses of generative AI is bound to be porn production, the latter may actually be pretty close to the truth: when a porn conosewer can’t tell the difference between AI generated video and video of the real thing, we will know that AGI (Artificial Genital Intelligence) has arrived.
But first, I suspect the AI engineers will have to fix the inevitable random swapping of male and female body parts in their generated video .
Or maybe not.
It's not only a massive waste of time the intellectual rot it has spawned has reached scientific publishing.
In their paper the authors wrote:
"Our findings suggest that the N-terminus of SN1 may act as a nucleation site for SNARE acceptor complex assembly. As a side note, our result of an intrinsic disorder of SNAP25 contrasts with a recent prediction of AlphaFold2 that predicted SN1 and SN2 to be entirely α-helical with high confidence." [1]
Interestingly the last sentence was eliminated in the published paper. Elsevier sells a number of AI journals so any well-founded criticism threatens one of their profit streams. Even mild criticism of AI is, thus, not allowed.
[1] Stief T, Gremer L, Pribicevic S, Espinueva DF, Vormann K, Biehl R, Jahn R, Pérez-Lara Á, Lakomek NA. Intrinsic Disorder of the Neuronal SNARE Protein SNAP25a in its Pre-fusion Conformation. J Mol Biol. 2023 May 15;435(10):168069. doi: 10.1016/j.jmb.2023.168069. Epub 2023 Mar 30. PMID: 37003471.
Yeah, journalists call that "balance". They did the same thing with climate change denial. Give one side to people who have no clue what they are talking about, and the other side to people who do.
They have zero clue what actual objectivity, or balance is. Its just the truth, nothing more. Its not entirely journalists fault though, given the corruption of data scientists in the tech industry. They don't know what the science is. They do know that they should seek out all scientific opinion and get a good sense of it.
I listened to Gary Marcus on radio new zealand, and it was a fine interview. But they just had to put an AI doomer on nextweek to balance it out. It bothers me because that radio station is not even commercialized. Its the last bastion of media that isn't corporate or capitalist controlled, or runs for profit.
You have a good point but I started to discover young developers with a new approach such https://www.cursor.com/
I discovered them both by the Lex Fridman podcast and I discovered Elliott who’s 20 and has a similar approach. They’re all extremely young but it’s interesting to listen to them and it’s very easy to contact them. Moreover, I think Elon never tried to censor you: you clashed about politics and you attacked the Tesla Robotaxi event: let it go, it’s just marketing. Please come back in the Adrian space. Thanks.
"systems that traffic only in the statistics of language without explicit representation of facts"
EXACTLY. Been saying this repeatedly myself. If something doesn't deal with truth values then how could anything "reason" with them? https://davidhsing.substack.com/p/why-neural-networks-is-a-bad-technology
Human's are all too happy to ignore facts these days in favor of what they want to believe.
Well written. Speaking as a student of technological change, this is not surprising. Nor is it surprising that legions of fans deny it!
All exponential improvement curves eventually turn into S curves (i.e. slowing rate of improvement) as one or more fundamental limits start to bear. They can last for a long time - look at multiple versions of Moore’s Law lasting 50+ years! But they cannot go on forever. (“If something cannot last forever, it won’t. I forget which SF author coined that.)
Fortunately for technology, another approach can eventually become relevant and its performance can surpass the old one. So the overall rate of improvement can look vaguely exponential, even though it’s a series of S curves. In fact LLMs in some ways were an example - a very different approach to AI rocketed past machine learning approaches, for some applications.
Of course this pattern is not proof, by itself, that LLMs are slowing down. But if not now, they will eventually. Others have written about why that will happen before “General AI” levels of performance.
However, as per OpenAIs own research, transformer based LLMs had already started to plateau in performance around gpt3.5. These models don’t reason, they memorize word patterns and act as extremely efficient Markov chain models. And there is no reason to believe that such a model , no matter how large the training data or parameter size will ever be able to reason. Still, as tools and simple question answer machines, they are very useful. They actually make us marvel at the awesomeness of brains and neurons, which can do so much with a fraction of the energy consumption of these constructs.
I see the quote is credited to economist Herbert Stein: https://en.wikipedia.org/wiki/Herbert_Stein
It’s not even a question of S curves though. Otherwise intelligent people literally believed that human intelligence was a 1950s-era neural net model merely accelerated with enough GPU cycle speeds and a massive data set.
Talk about riding reductionism off a Wile E. Coyote cliff.
May be off a slightly different tangent...I get the feeling it's a similar kind of operationalism that Watson, later Skinner introduced. The kind of operationalism that is also part of Steven's measurement model that dominates social sciences: as long as you map the empirical relations to numerical ones more or less consistently, you have measured. Measured what exactly? Doesn't matter, the model does not say, who cares? Turing's eponymous test is of the same kind, not even validated, just asserted. As long as current AI activities live in that bubble nothing will change. Even if the bubble bursts, nothing will change. "Let's not dwell on the past, let's move forward...". There won't be a truth commission. Current AI is good enough a pretext to fire people, hire younger ones at lower salaries in countries with lower labor regulations. Bottomline improves, profit improves, almost everyone is happy. And for all the expenses and unused graphic cards, there will be special tax regulations to ease the burden on the industry. After all, everybody was wrong, it was only business, not personal.
They marketed AGI and ASI, raised funds based on that and delivered amazing, smart indices.
So, it sounds like another Theranos situation where the media never bother to really talk/listen to the pathologists (or, in this case the scientists/researchers in the AI trenches) and instead only listen to those with a vested interest.
I think the Theranos comparison is unfair. Theranos committed outright fraud. I don't hear the AI companies being accused of that. As far as I know, there's no law against unrealistic expectations or simply bad reasoning. Many companies fail because the stuff they're working on doesn't fulfill expectations. It's not fraud.
I largely agree with you. I mostly don't believe anyone is out and out perpetuating fraud. Still, when I hear Sam Altman or (in the past) Mira Murati claim in interviews that they're seeing signs of "intelligence" " emerging" from their next-gen models, I have to wonder. Could they possibly believe this? Obviously they have a lot of financial incentive to drink their own Kool-Aid. Do they believe it in the abstract and feel like they need to buy more time to make it happen? I don't know how to interpret it.
from my talking to people working at places like openAI, they truly believe it
Who knows what they truly believe? It doesn't really matter because they are free to hold whatever opinions they want. It is usual for CEOs and CTOs to hype their products. It is on us to decide whether what they say is useful, with the help of experts like Gary Marcus of course.
Until the algorithms change radically from the LLM technology they are all using now, I would dismiss all comments about "intelligence emerging". Whatever intelligence they output is derived from the human intelligence embodied in their training data, which is pretty much the entire internet. They are improving their products by scaling (ie, more training data) and little tweaks to try to control their output (eg, adhere to social norms, filter in favor of truth). As far as I can tell, they have no plan to reach AGI other than a general desire to do so.
You are right, of course. However, I could care less whether its fraud or people choosing to believe whatever they want.
If people who have all the money in the world and every opportunity to listen to criticism want to lie to themselves, and it causes a lot of harm in the process, then its just a clever way of gaming the system. Fraud at least allows them to be put in jail more easily.
I'd prefer to just follow my moral intuitions here than accept some arbitrary law.
Aside from that, LLMs/AI is a much bigger industry than Theranos, which was small potatoes compared to this really.
Fair - I think it wasn't Theranos - but might be about to tip over into that. If Open AI et al. have discovered what they are selling (GAI) is bunk, then they will be commiting fraud when they go to the market and sell it as such.
Certainly there is motivation to do fraud when there's so much money riding on it. On the other hand, the nature of AI and LLMs is that a company has to release it into the wild before people will regard it as real, regardless of their talk. That makes it hard to commit fraud.
Unrealistic expectations in and of itself are not fraud. However, selling those unrealistic expectations while you know they can't be met in any way, is. That is precisely what Theranos was doing, selling something they knew they couldn't actually deliver. I fail to see the difference in what OpenAI is selling to the world. And if the OpenAI crowd isn't aware of this, then they are merely delusional.
I believe Theranos’ fraud was telling people their blood was being processed by their invention while actually sending it to a regular lab. They stepped over the line. As far as I know, OpenAI hasn’t done anything like this. What they are actually selling can be tried out by their customers. Telling people that AGI is just around the corner is just hype and legal. Of course, if it is seen as unrealistic, it will damage their reputation and investors will bail. I think we are at that point now.
Actually, she was not convicted for that specifically, she was convicted for telling investors she could do something that she knowingly couldn't. The big debate, after the whole Theranos debacle, was about the Silicon Valley idea of "fake it until you make it". OpenAI is also telling investors they can do something (in the near future) that they know they cannot. The only difference with Theranos is that the actual risk for people using it, is harder to understand (fake blood-testing results vs fake information).
Regardless of whether a particular case rises to the level of fraud in the legal sense, one thing is clear: the “fake it till you make it” crowd give legitimate scientists and engineers a bad name.
No self respecting scientist could/would knowingly work for an organization that was engaging in that.
I knew LeCun initially panned you and then slyly realigned himself closer to you (I remember asking on an earlier article of yours if he'd come out and admitted as much) but this is the first I'm hearing of him trying to get you deplatformed. What an odious creature.
at one point he blatantly lied about my credentials and refused to correct (you can read about it at ZDnet). odious is the correct word.
by, inter alia, lying about my credentials in a ZDNet article, and refusing to correct, even when I showed he was definitely wrong.
i wouldn’t say something like that if I didn’t have the goods. (i did write about it in a substack before btw)
Then ask Gary to write one.
LeCun never responded to that specific essay, but yes he did call me a grifter, and I blocked him for the ad hominem attack. he blocked me back. i have offered a truce; he has refused to respond.
please don’t lie Andersen. what you said is wrong, and leaves out the lying about credentials which is public.
fix what you said or bye bye
OK.
If you rewrote this article _without_ the _eighteen_ uses of I, me, and my, people might take it more seriously. Otherwise some might think that this is entirely an ego exercise on your part.
You ever been wrongly attacked by Altman, musk, and Leacin and hundred more for saying something that is true?
No, I haven't. And that is undoubtedly infuriating. However, I still think that a less personal response would carry more weight with readers.
I think you are asking too much! Find one person who takes this much beating from grifters and swindlers and are still standing and churning out post after post. I am sure for many individuals in the industry Gary's non-existence is preferrable to his existence. It's easy for you to preach for calm and balanced tone when you are not the one in the ring fighting.
When the attacks go to one's character/integrity/intelligence/quality of work...that is inherently personal. I feel for you...as I think that anyone paying attention must conclude that you have been a public beacon of intellectual honesty and integrity...and that others/biz have not...for obvious reasons ($$/power). And using 'it's only business, nothing personal'...often said with a wink...is still a cruel joke for those of us that try to remain human.
See that is interesting. When I read this Gary's article first time I did not even notice those "I, me, my" etc. I simply got a timeline of Gary's predictions and subsequent findings. After I read your post and went back to read the same article again, those "I, me, my" still did not catch my attention. See I pay attention to what were said, not how they were said, because they are irrelevant.
Very well said. Gary ruins what intellect he has with constant self-promoting, self-congratulatory and egotistical rants.
If what he stated were facts, what does it matter if it appears self-promoting or not to someone? In that case it is that someone's problem not his. If what he stated were not facts, then point out those non-facts please! Don't just put out "opinion" without evidence.
Inevitable plateau. We could say the same about large image classifier and object detector models based on monolithic 'lets just add more data in even bigger networks' CNNs and R-CNNs. But, as more hybridisation takes place (e.g. adding logic and automated reasoning branches of AI, multi agent solutions, etc.) their capabilities will rise again. No longer LLMs then? Cheating? Who cares.
LLMs have been pushed into applications which go beyond their language capabilities. Inevitable. But hybridisation and or integration and systems engineering methods can come to the rescue to make them work well enough for more albeit not all applications.
Opinion only.
Thanks for being a voice of reason. My guess is that we reached a stagnation phase similar to the one we saw with a lot of other technologies that were thought to be on the verge of a major breakthrough before things slowed down.
Brilliant and timely piece bringing up Marc Andreesen’s observation.
We are with you Gary on stating over and over what others will painfully discover.
Also, we are working on other approaches to autonomous agents beyond transformer-based GenAI, I hope more people do too.
Thanks for your input Andy.
Transformer-based LLMs are definitely something to remember as we move forward with AI R&D, but they should not be a stepping stone; this is my opinion and what I am practicing.
Learning about Transformers gave me an idea to algorithmically correlate words to each other.
The simplest useful relationship can be expressed as a trigram: previous word, current word, and next word.
Using those trigrams, a Natural Language Generator (NLG) that I wrote makes grammatically-correct sentences not in the training set such as the ones below:
====== NLG-generated ==========
I want to speak more.
I want to change her tune.
================
There is too much overhead and uncertainty in GenAI's Transformers to make them a foundation for future AI R&D.
Like Gary says, LLM will have their place in the AI pantheon, but not as a stepping stone to any kind of true AI, even less AGI.
Hi Andy, we can fuse that vast amount of data and make statistical predictions using algorithms running on the C and other computer languages.
That is exactly what I am doing in my NLG.
I trained it with 13 public domain books, I coded its algorithms, I can explain how it generates each sentence.
We can map the capabilities of Transformers into much more efficient programs.
I trained and inference on a 10-year old laptop running Linux and using no more than 2Gbytes of memory and no more than 200 MBytes of disk.
Plus, I don't need to start over, I've been working on my AI R&D for a year now.
With apologies to Shakespeare:
Double, double AI trouble
Billion$ burned in finance bubble.
Oil from an AI snake,
In the caldron boil and bake;
GPT and toe of Grok
Gemini and Llama talk
Adder's fork and blind-worm's sting,
Lizard's leg and howlet's wing,
For a charm of powerful trouble,
Like a hell-broth boil and bubble.
Double, double AI trouble
Billion$ burned in finance bubble.
Cool it with investors’ blood,
Then the charm is firm and good.
“systems that traffic only in the statistics of language without explicit representation of facts"
And CEOs that traffic only in the hyperbole of language without explicit representation of facts (Lying Liar Models)
This article has _lowered_ my estimate of probability of LLMs hitting the wall.
First, there's very little in terms of substance and very much in terms of "I told you so" and accusing others of being alchemists. (What's next? LLM as phlogiston?)
Second, it quotes Marc Andreesen, which is a sure source for "listen and do the reverse".
ha ha. some of the publicly available data were in the first link, of course.
The first link? The 2022 opinion piece on how we need to solve hallucination and abstraction to reason?
I'll be generous and presume, deducing it from your other comment, that you meant the _second_ link. It is indeed an argument for the trend being less exponential that some expect. (Not an utterly unbeatable argument, mind you, but an argument nonetheless) But if the growth becomes, say, "merely" quadratic, it will still deliver quite some results for years to come.
Would you please link to the empirical evidence?
the initial evidence was linked at top near the word April. a lot of recent evidence is proprietary, leaking out, but clearly consistent with what i anticipated.
I appreciate your perspective in these times of AI hype. What companies or research groups do you think are making the most interesting progress in synthesizing logic/reasoning with reference data to excel where LLMs fall short? Do the most promising approaches even involve neural nets?
I can't think of a time when public figures have been so publicly confident of something that's pure faith. It's more akin to religion than science! And yet they will shout you down so fast if you point to the naked emperor.