Gary, I heard you list 10 great reasons for skepticism on the BBC. The interviewer acknowledged you with mild interest and then gave equal time to an investor who shared hype and zero counter evidence.
Meanwhile, the hype has every business searching for use cases, most of which are pretty meh. An AI Assistant is just a form, built with trial and error, to reduce trial and error for someone building a prompt. And the output still likely includes errors. And the Assistant works okay for a few months, and then stops working, and the people who created it don’t know why. What a massive waste of time.
Although I am also an LLM skeptic, I do want to point out that it is hard for investors and AI gurus to offer anything approaching evidence for their enthusiasm. They think, or pretend to think, that their technology will get better by future scaling and/or unspecified incremental algorithm improvement. It is very difficult to prove that they can't get there. Still, I have great confidence that they won't.
Interesting thought! Isn’t it generally harder to prove a negative? I heard a safety head from open AI insisting that AGI is coming very soon, and then mentioned that one of his concerns is that no one is paying any attention to solving hallucinations. The contradiction in those claims is amazing. Sure, all of his financial incentives are telling him to ignore it.
I have no opinion on the long term for AI. It will be here in some form, for good or ill, unless there is a massive backlash to the harms.
Yeah, journalists call that "balance". They did the same thing with climate change denial. Give one side to people who have no clue what they are talking about, and the other side to people who do.
They have zero clue what actual objectivity, or balance is. Its just the truth, nothing more. Its not entirely journalists fault though, given the corruption of data scientists in the tech industry. They don't know what the science is. They do know that they should seek out all scientific opinion and get a good sense of it.
I listened to Gary Marcus on radio new zealand, and it was a fine interview. But they just had to put an AI doomer on nextweek to balance it out. It bothers me because that radio station is not even commercialized. Its the last bastion of media that isn't corporate or capitalist controlled, or runs for profit.
You have a good point but I started to discover young developers with a new approach such https://www.cursor.com/
I discovered them both by the Lex Fridman podcast and I discovered Elliott who’s 20 and has a similar approach. They’re all extremely young but it’s interesting to listen to them and it’s very easy to contact them. Moreover, I think Elon never tried to censor you: you clashed about politics and you attacked the Tesla Robotaxi event: let it go, it’s just marketing. Please come back in the Adrian space. Thanks.
Well written. Speaking as a student of technological change, this is not surprising. Nor is it surprising that legions of fans deny it!
All exponential improvement curves eventually turn into S curves (i.e. slowing rate of improvement) as one or more fundamental limits start to bear. They can last for a long time - look at multiple versions of Moore’s Law lasting 50+ years! But they cannot go on forever. (“If something cannot last forever, it won’t. I forget which SF author coined that.)
Fortunately for technology, another approach can eventually become relevant and its performance can surpass the old one. So the overall rate of improvement can look vaguely exponential, even though it’s a series of S curves. In fact LLMs in some ways were an example - a very different approach to AI rocketed past machine learning approaches, for some applications.
Of course this pattern is not proof, by itself, that LLMs are slowing down. But if not now, they will eventually. Others have written about why that will happen before “General AI” levels of performance.
However, as per OpenAIs own research, transformer based LLMs had already started to plateau in performance around gpt3.5. These models don’t reason, they memorize word patterns and act as extremely efficient Markov chain models. And there is no reason to believe that such a model , no matter how large the training data or parameter size will ever be able to reason. Still, as tools and simple question answer machines, they are very useful. They actually make us marvel at the awesomeness of brains and neurons, which can do so much with a fraction of the energy consumption of these constructs.
It’s not even a question of S curves though. Otherwise intelligent people literally believed that human intelligence was a 1950s-era neural net model merely accelerated with enough GPU cycle speeds and a massive data set.
Talk about riding reductionism off a Wile E. Coyote cliff.
May be off a slightly different tangent...I get the feeling it's a similar kind of operationalism that Watson, later Skinner introduced. The kind of operationalism that is also part of Steven's measurement model that dominates social sciences: as long as you map the empirical relations to numerical ones more or less consistently, you have measured. Measured what exactly? Doesn't matter, the model does not say, who cares? Turing's eponymous test is of the same kind, not even validated, just asserted. As long as current AI activities live in that bubble nothing will change. Even if the bubble bursts, nothing will change. "Let's not dwell on the past, let's move forward...". There won't be a truth commission. Current AI is good enough a pretext to fire people, hire younger ones at lower salaries in countries with lower labor regulations. Bottomline improves, profit improves, almost everyone is happy. And for all the expenses and unused graphic cards, there will be special tax regulations to ease the burden on the industry. After all, everybody was wrong, it was only business, not personal.
So, it sounds like another Theranos situation where the media never bother to really talk/listen to the pathologists (or, in this case the scientists/researchers in the AI trenches) and instead only listen to those with a vested interest.
I think the Theranos comparison is unfair. Theranos committed outright fraud. I don't hear the AI companies being accused of that. As far as I know, there's no law against unrealistic expectations or simply bad reasoning. Many companies fail because the stuff they're working on doesn't fulfill expectations. It's not fraud.
I largely agree with you. I mostly don't believe anyone is out and out perpetuating fraud. Still, when I hear Sam Altman or (in the past) Mira Murati claim in interviews that they're seeing signs of "intelligence" " emerging" from their next-gen models, I have to wonder. Could they possibly believe this? Obviously they have a lot of financial incentive to drink their own Kool-Aid. Do they believe it in the abstract and feel like they need to buy more time to make it happen? I don't know how to interpret it.
Who knows what they truly believe? It doesn't really matter because they are free to hold whatever opinions they want. It is usual for CEOs and CTOs to hype their products. It is on us to decide whether what they say is useful, with the help of experts like Gary Marcus of course.
Until the algorithms change radically from the LLM technology they are all using now, I would dismiss all comments about "intelligence emerging". Whatever intelligence they output is derived from the human intelligence embodied in their training data, which is pretty much the entire internet. They are improving their products by scaling (ie, more training data) and little tweaks to try to control their output (eg, adhere to social norms, filter in favor of truth). As far as I can tell, they have no plan to reach AGI other than a general desire to do so.
You are right, of course. However, I could care less whether its fraud or people choosing to believe whatever they want.
If people who have all the money in the world and every opportunity to listen to criticism want to lie to themselves, and it causes a lot of harm in the process, then its just a clever way of gaming the system. Fraud at least allows them to be put in jail more easily.
I'd prefer to just follow my moral intuitions here than accept some arbitrary law.
Aside from that, LLMs/AI is a much bigger industry than Theranos, which was small potatoes compared to this really.
Fair - I think it wasn't Theranos - but might be about to tip over into that. If Open AI et al. have discovered what they are selling (GAI) is bunk, then they will be commiting fraud when they go to the market and sell it as such.
Certainly there is motivation to do fraud when there's so much money riding on it. On the other hand, the nature of AI and LLMs is that a company has to release it into the wild before people will regard it as real, regardless of their talk. That makes it hard to commit fraud.
Unrealistic expectations in and of itself are not fraud. However, selling those unrealistic expectations while you know they can't be met in any way, is. That is precisely what Theranos was doing, selling something they knew they couldn't actually deliver. I fail to see the difference in what OpenAI is selling to the world. And if the OpenAI crowd isn't aware of this, then they are merely delusional.
I believe Theranos’ fraud was telling people their blood was being processed by their invention while actually sending it to a regular lab. They stepped over the line. As far as I know, OpenAI hasn’t done anything like this. What they are actually selling can be tried out by their customers. Telling people that AGI is just around the corner is just hype and legal. Of course, if it is seen as unrealistic, it will damage their reputation and investors will bail. I think we are at that point now.
Actually, she was not convicted for that specifically, she was convicted for telling investors she could do something that she knowingly couldn't. The big debate, after the whole Theranos debacle, was about the Silicon Valley idea of "fake it until you make it". OpenAI is also telling investors they can do something (in the near future) that they know they cannot. The only difference with Theranos is that the actual risk for people using it, is harder to understand (fake blood-testing results vs fake information).
Regardless of whether a particular case rises to the level of fraud in the legal sense, one thing is clear: the “fake it till you make it” crowd give legitimate scientists and engineers a bad name.
I knew LeCun initially panned you and then slyly realigned himself closer to you (I remember asking on an earlier article of yours if he'd come out and admitted as much) but this is the first I'm hearing of him trying to get you deplatformed. What an odious creature.
Lying about your credentials has nothing to do with de-platforming and has nothing to do with your prediction on the future of LLM. You are conflating 3 unrelated issues here.
If your goal is to talk about your prediction of LLM's demise, please focus on that. Sound critical analysis is what provides value, not unrelated personal issues.
LeCun and Gary have been sparing as folks always do, till Gary called LeCun one of the five most dangerous people in tech, which was, in my view, a terrible and deeply unfair take, LeCun responded not too nicely, then got each other blocked.
If that's what "de-platforming" means, then this is again the usual inaccurate polemics and silly innuendo.
LeCun never responded to that specific essay, but yes he did call me a grifter, and I blocked him for the ad hominem attack. he blocked me back. i have offered a truce; he has refused to respond.
Inevitable plateau. We could say the same about large image classifier and object detector models based on monolithic 'lets just add more data in even bigger networks' CNNs and R-CNNs. But, as more hybridisation takes place (e.g. adding logic and automated reasoning branches of AI, multi agent solutions, etc.) their capabilities will rise again. No longer LLMs then? Cheating? Who cares.
LLMs have been pushed into applications which go beyond their language capabilities. Inevitable. But hybridisation and or integration and systems engineering methods can come to the rescue to make them work well enough for more albeit not all applications.
Thanks for being a voice of reason. My guess is that we reached a stagnation phase similar to the one we saw with a lot of other technologies that were thought to be on the verge of a major breakthrough before things slowed down.
If you rewrote this article _without_ the _eighteen_ uses of I, me, and my, people might take it more seriously. Otherwise some might think that this is entirely an ego exercise on your part.
When the attacks go to one's character/integrity/intelligence/quality of work...that is inherently personal. I feel for you...as I think that anyone paying attention must conclude that you have been a public beacon of intellectual honesty and integrity...and that others/biz have not...for obvious reasons ($$/power). And using 'it's only business, nothing personal'...often said with a wink...is still a cruel joke for those of us that try to remain human.
Starting over does not make any sense. LLM are very good distillators and integrators of data. They will be the foundation on which intelligent agents are built. Such an agents will need many other parts, including domain-specific strategies for modeling and verification.
"explicit representation of facts and explicit tools to reason over those facts." would be very nice, but are very hard to do. The approach on leaning as much as possible on neural nets for doing it implicitly has been working a lot better for the last 20 years.
Transformer-based LLMs are definitely something to remember as we move forward with AI R&D, but they should not be a stepping stone; this is my opinion and what I am practicing.
Learning about Transformers gave me an idea to algorithmically correlate words to each other.
The simplest useful relationship can be expressed as a trigram: previous word, current word, and next word.
Using those trigrams, a Natural Language Generator (NLG) that I wrote makes grammatically-correct sentences not in the training set such as the ones below:
====== NLG-generated ==========
I want to speak more.
I want to change her tune.
================
There is too much overhead and uncertainty in GenAI's Transformers to make them a foundation for future AI R&D.
This article has _lowered_ my estimate of probability of LLMs hitting the wall.
First, there's very little in terms of substance and very much in terms of "I told you so" and accusing others of being alchemists. (What's next? LLM as phlogiston?)
Second, it quotes Marc Andreesen, which is a sure source for "listen and do the reverse".
The first link? The 2022 opinion piece on how we need to solve hallucination and abstraction to reason?
I'll be generous and presume, deducing it from your other comment, that you meant the _second_ link. It is indeed an argument for the trend being less exponential that some expect. (Not an utterly unbeatable argument, mind you, but an argument nonetheless) But if the growth becomes, say, "merely" quadratic, it will still deliver quite some results for years to come.
the initial evidence was linked at top near the word April. a lot of recent evidence is proprietary, leaking out, but clearly consistent with what i anticipated.
I appreciate your perspective in these times of AI hype. What companies or research groups do you think are making the most interesting progress in synthesizing logic/reasoning with reference data to excel where LLMs fall short? Do the most promising approaches even involve neural nets?
I can't think of a time when public figures have been so publicly confident of something that's pure faith. It's more akin to religion than science! And yet they will shout you down so fast if you point to the naked emperor.
Scaling ain't just one thing. We are seeing that in greater emphasis in post training. Tasks are accomplished stepwise. Functionally that is chains of sigmoids. As flexible architectures better exploit sigmoid activations we will enter the domain of right size scaling.
Gary, I heard you list 10 great reasons for skepticism on the BBC. The interviewer acknowledged you with mild interest and then gave equal time to an investor who shared hype and zero counter evidence.
Meanwhile, the hype has every business searching for use cases, most of which are pretty meh. An AI Assistant is just a form, built with trial and error, to reduce trial and error for someone building a prompt. And the output still likely includes errors. And the Assistant works okay for a few months, and then stops working, and the people who created it don’t know why. What a massive waste of time.
your first paragraph is so damning, and a tremendous illustration of what went wrong. i will quote it sometime!
Although I am also an LLM skeptic, I do want to point out that it is hard for investors and AI gurus to offer anything approaching evidence for their enthusiasm. They think, or pretend to think, that their technology will get better by future scaling and/or unspecified incremental algorithm improvement. It is very difficult to prove that they can't get there. Still, I have great confidence that they won't.
Interesting thought! Isn’t it generally harder to prove a negative? I heard a safety head from open AI insisting that AGI is coming very soon, and then mentioned that one of his concerns is that no one is paying any attention to solving hallucinations. The contradiction in those claims is amazing. Sure, all of his financial incentives are telling him to ignore it.
I have no opinion on the long term for AI. It will be here in some form, for good or ill, unless there is a massive backlash to the harms.
Yeah, journalists call that "balance". They did the same thing with climate change denial. Give one side to people who have no clue what they are talking about, and the other side to people who do.
They have zero clue what actual objectivity, or balance is. Its just the truth, nothing more. Its not entirely journalists fault though, given the corruption of data scientists in the tech industry. They don't know what the science is. They do know that they should seek out all scientific opinion and get a good sense of it.
I listened to Gary Marcus on radio new zealand, and it was a fine interview. But they just had to put an AI doomer on nextweek to balance it out. It bothers me because that radio station is not even commercialized. Its the last bastion of media that isn't corporate or capitalist controlled, or runs for profit.
You have a good point but I started to discover young developers with a new approach such https://www.cursor.com/
I discovered them both by the Lex Fridman podcast and I discovered Elliott who’s 20 and has a similar approach. They’re all extremely young but it’s interesting to listen to them and it’s very easy to contact them. Moreover, I think Elon never tried to censor you: you clashed about politics and you attacked the Tesla Robotaxi event: let it go, it’s just marketing. Please come back in the Adrian space. Thanks.
Well written. Speaking as a student of technological change, this is not surprising. Nor is it surprising that legions of fans deny it!
All exponential improvement curves eventually turn into S curves (i.e. slowing rate of improvement) as one or more fundamental limits start to bear. They can last for a long time - look at multiple versions of Moore’s Law lasting 50+ years! But they cannot go on forever. (“If something cannot last forever, it won’t. I forget which SF author coined that.)
Fortunately for technology, another approach can eventually become relevant and its performance can surpass the old one. So the overall rate of improvement can look vaguely exponential, even though it’s a series of S curves. In fact LLMs in some ways were an example - a very different approach to AI rocketed past machine learning approaches, for some applications.
Of course this pattern is not proof, by itself, that LLMs are slowing down. But if not now, they will eventually. Others have written about why that will happen before “General AI” levels of performance.
However, as per OpenAIs own research, transformer based LLMs had already started to plateau in performance around gpt3.5. These models don’t reason, they memorize word patterns and act as extremely efficient Markov chain models. And there is no reason to believe that such a model , no matter how large the training data or parameter size will ever be able to reason. Still, as tools and simple question answer machines, they are very useful. They actually make us marvel at the awesomeness of brains and neurons, which can do so much with a fraction of the energy consumption of these constructs.
I see the quote is credited to economist Herbert Stein: https://en.wikipedia.org/wiki/Herbert_Stein
It’s not even a question of S curves though. Otherwise intelligent people literally believed that human intelligence was a 1950s-era neural net model merely accelerated with enough GPU cycle speeds and a massive data set.
Talk about riding reductionism off a Wile E. Coyote cliff.
May be off a slightly different tangent...I get the feeling it's a similar kind of operationalism that Watson, later Skinner introduced. The kind of operationalism that is also part of Steven's measurement model that dominates social sciences: as long as you map the empirical relations to numerical ones more or less consistently, you have measured. Measured what exactly? Doesn't matter, the model does not say, who cares? Turing's eponymous test is of the same kind, not even validated, just asserted. As long as current AI activities live in that bubble nothing will change. Even if the bubble bursts, nothing will change. "Let's not dwell on the past, let's move forward...". There won't be a truth commission. Current AI is good enough a pretext to fire people, hire younger ones at lower salaries in countries with lower labor regulations. Bottomline improves, profit improves, almost everyone is happy. And for all the expenses and unused graphic cards, there will be special tax regulations to ease the burden on the industry. After all, everybody was wrong, it was only business, not personal.
"systems that traffic only in the statistics of language without explicit representation of facts"
EXACTLY. Been saying this repeatedly myself. If something doesn't deal with truth values then how could anything "reason" with them? https://davidhsing.substack.com/p/why-neural-networks-is-a-bad-technology
They marketed AGI and ASI, raised funds based on that and delivered amazing, smart indices.
So, it sounds like another Theranos situation where the media never bother to really talk/listen to the pathologists (or, in this case the scientists/researchers in the AI trenches) and instead only listen to those with a vested interest.
I think the Theranos comparison is unfair. Theranos committed outright fraud. I don't hear the AI companies being accused of that. As far as I know, there's no law against unrealistic expectations or simply bad reasoning. Many companies fail because the stuff they're working on doesn't fulfill expectations. It's not fraud.
I largely agree with you. I mostly don't believe anyone is out and out perpetuating fraud. Still, when I hear Sam Altman or (in the past) Mira Murati claim in interviews that they're seeing signs of "intelligence" " emerging" from their next-gen models, I have to wonder. Could they possibly believe this? Obviously they have a lot of financial incentive to drink their own Kool-Aid. Do they believe it in the abstract and feel like they need to buy more time to make it happen? I don't know how to interpret it.
Who knows what they truly believe? It doesn't really matter because they are free to hold whatever opinions they want. It is usual for CEOs and CTOs to hype their products. It is on us to decide whether what they say is useful, with the help of experts like Gary Marcus of course.
Until the algorithms change radically from the LLM technology they are all using now, I would dismiss all comments about "intelligence emerging". Whatever intelligence they output is derived from the human intelligence embodied in their training data, which is pretty much the entire internet. They are improving their products by scaling (ie, more training data) and little tweaks to try to control their output (eg, adhere to social norms, filter in favor of truth). As far as I can tell, they have no plan to reach AGI other than a general desire to do so.
You are right, of course. However, I could care less whether its fraud or people choosing to believe whatever they want.
If people who have all the money in the world and every opportunity to listen to criticism want to lie to themselves, and it causes a lot of harm in the process, then its just a clever way of gaming the system. Fraud at least allows them to be put in jail more easily.
I'd prefer to just follow my moral intuitions here than accept some arbitrary law.
Aside from that, LLMs/AI is a much bigger industry than Theranos, which was small potatoes compared to this really.
Fair - I think it wasn't Theranos - but might be about to tip over into that. If Open AI et al. have discovered what they are selling (GAI) is bunk, then they will be commiting fraud when they go to the market and sell it as such.
Certainly there is motivation to do fraud when there's so much money riding on it. On the other hand, the nature of AI and LLMs is that a company has to release it into the wild before people will regard it as real, regardless of their talk. That makes it hard to commit fraud.
Unrealistic expectations in and of itself are not fraud. However, selling those unrealistic expectations while you know they can't be met in any way, is. That is precisely what Theranos was doing, selling something they knew they couldn't actually deliver. I fail to see the difference in what OpenAI is selling to the world. And if the OpenAI crowd isn't aware of this, then they are merely delusional.
I believe Theranos’ fraud was telling people their blood was being processed by their invention while actually sending it to a regular lab. They stepped over the line. As far as I know, OpenAI hasn’t done anything like this. What they are actually selling can be tried out by their customers. Telling people that AGI is just around the corner is just hype and legal. Of course, if it is seen as unrealistic, it will damage their reputation and investors will bail. I think we are at that point now.
Actually, she was not convicted for that specifically, she was convicted for telling investors she could do something that she knowingly couldn't. The big debate, after the whole Theranos debacle, was about the Silicon Valley idea of "fake it until you make it". OpenAI is also telling investors they can do something (in the near future) that they know they cannot. The only difference with Theranos is that the actual risk for people using it, is harder to understand (fake blood-testing results vs fake information).
Regardless of whether a particular case rises to the level of fraud in the legal sense, one thing is clear: the “fake it till you make it” crowd give legitimate scientists and engineers a bad name.
No self respecting scientist could/would knowingly work for an organization that was engaging in that.
I knew LeCun initially panned you and then slyly realigned himself closer to you (I remember asking on an earlier article of yours if he'd come out and admitted as much) but this is the first I'm hearing of him trying to get you deplatformed. What an odious creature.
at one point he blatantly lied about my credentials and refused to correct (you can read about it at ZDnet). odious is the correct word.
"have tried to deplatform me."
How is that LeCun tried to de-platform Gary? I want actual solid evidence, not innuendo.
I would be nice to read an article that seriously analyzes the state of the industry, rather than the usual list of grievances.
by, inter alia, lying about my credentials in a ZDNet article, and refusing to correct, even when I showed he was definitely wrong.
i wouldn’t say something like that if I didn’t have the goods. (i did write about it in a substack before btw)
Lying about your credentials has nothing to do with de-platforming and has nothing to do with your prediction on the future of LLM. You are conflating 3 unrelated issues here.
If your goal is to talk about your prediction of LLM's demise, please focus on that. Sound critical analysis is what provides value, not unrelated personal issues.
Then ask Gary to write one.
LeCun and Gary have been sparing as folks always do, till Gary called LeCun one of the five most dangerous people in tech, which was, in my view, a terrible and deeply unfair take, LeCun responded not too nicely, then got each other blocked.
If that's what "de-platforming" means, then this is again the usual inaccurate polemics and silly innuendo.
LeCun never responded to that specific essay, but yes he did call me a grifter, and I blocked him for the ad hominem attack. he blocked me back. i have offered a truce; he has refused to respond.
please don’t lie Andersen. what you said is wrong, and leaves out the lying about credentials which is public.
fix what you said or bye bye
OK.
Inevitable plateau. We could say the same about large image classifier and object detector models based on monolithic 'lets just add more data in even bigger networks' CNNs and R-CNNs. But, as more hybridisation takes place (e.g. adding logic and automated reasoning branches of AI, multi agent solutions, etc.) their capabilities will rise again. No longer LLMs then? Cheating? Who cares.
LLMs have been pushed into applications which go beyond their language capabilities. Inevitable. But hybridisation and or integration and systems engineering methods can come to the rescue to make them work well enough for more albeit not all applications.
Opinion only.
Thanks for being a voice of reason. My guess is that we reached a stagnation phase similar to the one we saw with a lot of other technologies that were thought to be on the verge of a major breakthrough before things slowed down.
If you rewrote this article _without_ the _eighteen_ uses of I, me, and my, people might take it more seriously. Otherwise some might think that this is entirely an ego exercise on your part.
You ever been wrongly attacked by Altman, musk, and Leacin and hundred more for saying something that is true?
No, I haven't. And that is undoubtedly infuriating. However, I still think that a less personal response would carry more weight with readers.
When the attacks go to one's character/integrity/intelligence/quality of work...that is inherently personal. I feel for you...as I think that anyone paying attention must conclude that you have been a public beacon of intellectual honesty and integrity...and that others/biz have not...for obvious reasons ($$/power). And using 'it's only business, nothing personal'...often said with a wink...is still a cruel joke for those of us that try to remain human.
Brilliant and timely piece bringing up Marc Andreesen’s observation.
We are with you Gary on stating over and over what others will painfully discover.
Also, we are working on other approaches to autonomous agents beyond transformer-based GenAI, I hope more people do too.
Starting over does not make any sense. LLM are very good distillators and integrators of data. They will be the foundation on which intelligent agents are built. Such an agents will need many other parts, including domain-specific strategies for modeling and verification.
"explicit representation of facts and explicit tools to reason over those facts." would be very nice, but are very hard to do. The approach on leaning as much as possible on neural nets for doing it implicitly has been working a lot better for the last 20 years.
Thanks for your input Andy.
Transformer-based LLMs are definitely something to remember as we move forward with AI R&D, but they should not be a stepping stone; this is my opinion and what I am practicing.
Learning about Transformers gave me an idea to algorithmically correlate words to each other.
The simplest useful relationship can be expressed as a trigram: previous word, current word, and next word.
Using those trigrams, a Natural Language Generator (NLG) that I wrote makes grammatically-correct sentences not in the training set such as the ones below:
====== NLG-generated ==========
I want to speak more.
I want to change her tune.
================
There is too much overhead and uncertainty in GenAI's Transformers to make them a foundation for future AI R&D.
People have been trying to do trigrams and other such things in linguistics for decades. It never scaled up. LLM revolutionized the field.
Like Gary says, LLM will have their place in the AI pantheon, but not as a stepping stone to any kind of true AI, even less AGI.
LLM is very good at fusing a vast amount of data and then producing a statistical prediction. No other machinery comes even close to doing that.
I think this will be the Swiss Army knife. As part of a larger system that can do things that LLM cannot.
Hi Andy, we can fuse that vast amount of data and make statistical predictions using algorithms running on the C and other computer languages.
That is exactly what I am doing in my NLG.
I trained it with 13 public domain books, I coded its algorithms, I can explain how it generates each sentence.
We can map the capabilities of Transformers into much more efficient programs.
I trained and inference on a 10-year old laptop running Linux and using no more than 2Gbytes of memory and no more than 200 MBytes of disk.
Plus, I don't need to start over, I've been working on my AI R&D for a year now.
With apologies to Shakespeare:
Double, double AI trouble
Billion$ burned in finance bubble.
Oil from an AI snake,
In the caldron boil and bake;
GPT and toe of Grok
Gemini and Llama talk
Adder's fork and blind-worm's sting,
Lizard's leg and howlet's wing,
For a charm of powerful trouble,
Like a hell-broth boil and bubble.
Double, double AI trouble
Billion$ burned in finance bubble.
Cool it with investors’ blood,
Then the charm is firm and good.
This article has _lowered_ my estimate of probability of LLMs hitting the wall.
First, there's very little in terms of substance and very much in terms of "I told you so" and accusing others of being alchemists. (What's next? LLM as phlogiston?)
Second, it quotes Marc Andreesen, which is a sure source for "listen and do the reverse".
ha ha. some of the publicly available data were in the first link, of course.
The first link? The 2022 opinion piece on how we need to solve hallucination and abstraction to reason?
I'll be generous and presume, deducing it from your other comment, that you meant the _second_ link. It is indeed an argument for the trend being less exponential that some expect. (Not an utterly unbeatable argument, mind you, but an argument nonetheless) But if the growth becomes, say, "merely" quadratic, it will still deliver quite some results for years to come.
Would you please link to the empirical evidence?
the initial evidence was linked at top near the word April. a lot of recent evidence is proprietary, leaking out, but clearly consistent with what i anticipated.
I appreciate your perspective in these times of AI hype. What companies or research groups do you think are making the most interesting progress in synthesizing logic/reasoning with reference data to excel where LLMs fall short? Do the most promising approaches even involve neural nets?
I can't think of a time when public figures have been so publicly confident of something that's pure faith. It's more akin to religion than science! And yet they will shout you down so fast if you point to the naked emperor.
Scaling ain't just one thing. We are seeing that in greater emphasis in post training. Tasks are accomplished stepwise. Functionally that is chains of sigmoids. As flexible architectures better exploit sigmoid activations we will enter the domain of right size scaling.