Thank you for citing and elevating a trusted source of business news that also always manages to humanize business stories, business data, and the ever perplexing gyrations of the economy. Long time fan of Markeplace, Kai and his team and his process. Enjoy your SubStack and its focus. Making ongoing sense of the rapidly evolving landscape with AI infiltrating as much as it is allowed together away with…can be exhausting and perplexing.
"bar" has three parts of speech, as a noun it has 20 meanings - what good is prediction by an LLM if it doesn't knoiw the meaning? The statement has the wrong basis.
Bringing GOFAI back in certain ways is good, but ultimately, we're stuck with arbitration instead of any sort of real "understanding" of any kind.
A very rough example of pseudocode to illustrate this arbitrary relationship:
let p="night"
input R
if R="day" then print p+"is"+R
Now, if I type "day", then the output would be "night is day". Great. Absolutely "correct output" according to its programming. It doesn’t necessarily "make sense" but it doesn’t have to because it’s the programming! The same goes with any other input that gets fed into the machine to produce output e.g., "nLc is auS", "e8jey is 3uD4", and so on.
To the machine, codes and inputs are nothing more than items and sequences to execute. There’s no meaning to this sequencing or execution activity to the machine. To the programmer, there is meaning because he or she conceptualizes and understands variables as representative placeholders of their conscious experiences. The machine doesn’t comprehend concepts such as "variables", "placeholders", "items", "sequences", "execution", etc. It just doesn’t comprehend, period. Thus, a machine never truly "knows" what it’s doing and can only take on the operational appearance of comprehension.
Due to what machines are, machines are stuck with pretend-representation and pretend-understanding. Machines are epistemically land-locked. There's no externality to them, they don't possess, and can't possess, intentionality (cf. term in Philosophy of Mind https://plato.stanford.edu/entries/intentionality/ ) as such, there's no such thing as actual grounding in machines.
I guess the (IMHO) false hope of current gen AI is that “knowing” pops up as an emergent property of a sufficiently large neural network. The hope is based on similarity between neurons in our brains and nodes in the artificial neural network. Individual neurons also do not “know”, neither do the proteins they are built from or metabolic processes that power the whole thing. Yet the brain (or the human organism as a whole) seems to “know”. Why I think the hope of getting this emergent property in a neural network is false? Because any neural network is a static non-linear mapping from inputs to outputs. “Knowing” (or if you will consciousness) requires temporal evolution and I believe also continuous learning and embodied existence (possibly also self reproduction to give it motivation to “do” anything). That seems to be, as of now, out of reach for our silicon based technology and synchronous state automatons that all current computers are. It may be out of reach of digital technology in general. Frankly, I hope it remains that way ;-).
One of the reasons I don’t expect any useful emergent properties from neural nets is that they’re really not at all like neurons in their functioning or evolution over time.
I doubt it will remain inaccessible to silicon forever. These things have “emerged” and been played with over evolution multiple times. I see no fundamental reason they need to be fully substrate dependent.
I think it’s fair to say silicon won’t naturally evolve this kind of intelligence, at least on the surface of the Earth.
But that’s a far cry from what biology as designer and tool user can achieve.
I’ve recently read a study making an argument that intelligence may not be computable. I unfortunately cannot find the link anymore :-(. It is possible that intelligence similar to human is not possible in a digital system due to the necessarily incomplete representation of reality caused by digitization. It still may be possible in an analog system. The other claim was that self-reproduction may be needed for any agency and consciousness.
I’d agree with that depending on what they define intelligence to be.
But I don’t know why intelligence similar to a human should be the goal at all? At least so far, no one has assumed the best way to get humans far is to build faster legs to attach to our hips.
Why should we build intelligence in a similarly bizarre fashion? That’s where I hope *computing* goes. Whether we end up calling it AI or whatever else.
Yes, I also don’t understand why AGI (pretty much regardless of definition) should by a worthy goal to pursue. I think we need technology to augment humans, not to emulate them. I don’t want any AI system to have independent opinion and goals.
If you happen to remember the title or author of the article I'd love to read it. I have been thinking about similar things and made a comment to that effect below - would love to learn what scholars of the subject are saying.
Neurons (and all other cells) do a truly massive amount of information processing about which we have only the most rudimentary knowledge.
Immune cells can learn to recognize and then kill viruses and bacteria.
While some might say this is really not “knowing”, how do they know that?
Lots of uncertainty but one thing is pretty clear: treating neurons as simple “weights” in a neural net is a gross oversimplification (at best) and just plain wrong (at worst).
You’re shifting goal posts, and in the process twisting my argument.
Again:
Life has agency
Sub parts of life have agency, going all the way down to where it’s just lifeless molecules. That is, the whole has a greater agency than the sum of its parts, but that doesn’t take away that the parts have agency.
Individual cells, especially do have agency. We also shape the fate of individual cells, but by “nature” and by choices we make, dietary, exposure, etc.
Individual cells can now be fitted with additional optogenetic proteins at the cell surface.
This means we can play with and control their agency.
We are also well on the path to designing other proteins and being able to integrate them into cells.
We can thus add design to life and control it.
We can design non life and control it.
Sub parts of our own body can and do go rogue and respond to signals “sub”-consciously , integrating signals that we consciously discount.
Either we’re a sum of agents who can go rogue, and this obviously changes what “life” or “self” means.
Or we can design agency that expands our own.
It’s a fine distinction but either way your hard line is obliterated by evidence. Machines can be designed to mimic life. Thus they can have agency the same way sub parts of life can have it.
Can they have equal agency to a human who is a sum of so many distinct cellular agents (more bacteria in the gut than your entire rest of the body… so you and I aren’t even genomically one thing)?
I doubt anytime soon. And I doubt we should want to do it. But I won’t say with certainty we *cannot* do it. Right now that’s an article of faith. Not factual science. Insisting it is means some other idiot can proceed with a design that screws me, and I’m a selfish piece of biology. I’d rather be on the drivers seat, not be the car.
I think Gary's point in this interview has been world models, rather than symbolic logic. So, first the words are given meaning, and only then any transformations are attempted.
"World models" translates to "what's programmed in." That's arbitrated. "Given meaning" is a hand-wave. It doesn't happen just because arbitration or matching happens.
I propose for now we defer the issue of how world models are implemented. The interview was very vague and focused on high level discussion of how to give machines world models and representations, including attempts by LeCun and Fei-Fei Li.
Of course there is bitter disagreement for how such "meaning" is to be put in, and if it is even possible at all.
It's a start, as we all know. However, LeCun as well as everyone else are still avoiding the "The Hard Problem of Machine Understanding" which is grounding. Somebody has to get to it... Otherwise every machine is going to be stuck at the behaviorist stage.
Is the “hard problem of machine understanding” a formal term? Seems to me it’s a restatement of the hard problem of consciousness, for machines, from the sound of things.
The hard problem of consciousness don't really have much to do with practical matters, while grounding is a technical issue that has to be eventually faked well enough to handle any task. Grounding doesn't have to be taken care of perfectly, just well enough that performance issues disappear. For example, satellites in orbit doesn't take into account many relativistic effects, yet still operate well enough so the omissions don't matter.
How to do grounding is a very long discussion. I happen to think that we have made very good progress with approximate means, with more to come, though I know folks as yourself will strenuously disagree, arguing it is dumb symbols all the way down.
As long as the performance eventually gets faked well enough, the theoretics can take a back seat. The trouble is all sorts of issues are rearing their ugly heads in practice, so "well enough" still isn't anywhere in sight.
Is it possible to agree with you that AGI is going to be hard, new architecture is needed and scaling is not all we need...
...and yet worry that Eliezer is right that if we do finally create AGI / ASI, that this will be our last invention.
Unless you can refute orthogonality / gradient descent and show how ASI could possibly be safe shouldnt you be saying "Don't make AGI" as your primary statement rather than telling people they are going about this wrong?
Amen. There should be no discussion of how to build AGI before there is overwhelming consensus agreement among all stakeholders (i.e., the entire human species) that AGI ought to be built in the first place. Me? I say "no."
There is a weird combination of faith, hubris and in some cases megalomania involved in the assumption that pursuing ASI is the proper goal for humanity.
The AI field is more a religion than a science. The practitioners behave like high priests (even Demi-gods), as if they are uniquely qualified and sanctioned to decide, speak and act for the rest of humanity.
And Then It Fell: Of the many branches of possible uses of broader AGI (as suggested in Gary's note) the worst is to imagine it in the hands of those who get off on surveillance, for any means or ends.
As I understand it, even if goods can come from it, and I'm sure someone will find something somewhere, it doesn't matter precisely because AGI is an article of intelligence; and insofar as raw intelligence, under the fuller application of human agency, can opt for the full spectrum of good and bad, articles of intelligence can be developed or accessed by an intelligent person acting under the influence of any part of that spectrum for any situational disturbance, for accidental or deliberate scenarios, and/or under conscious, partly conscious, or unconscious motivations.
. . . and we don't want to attack human agency as such, or the freedom side of the freedom-to-responsibility complex, or do we? And what would it mean to be a noble person, even if one could, if everyone else is under one's watchful eye or already dead? I could go on, but you probably get the picture.
Probably the only thing CERTAIN about the trajectory of AI is that it will take on an ever more (probably completely) encompassing surveillance role.
Even current AI capabilities would have brought members of the East German Stasi to orgasm. They would have spent every waking hour in the AI-gasmatron.
The mistake that many a proponent of AGI/ASI makes is assuming that risk depends only on the probability of an event.
The reality (as every insurance company knows) is that
Risk = probability X IMPACT
So even (assumed) low probability high impact events (eg, end of democracy, societal breakdown or even human extinction) can have high associated risk.(what “impact/cost” should we assign to human extinction, by the way? )
Many proponents of AGI try to dismiss concerns by stating that human extinction is highly unlikely (whatever “highly” means). While that is probably (albeit by no means certainly) the case, extinction is hardly the only possible high impact event (or even the one people should necessarily focus on)
And simply considering probability (which no one really has anything but the foggiest notion of, anyway) without factoring in impact renders all talk of AGI risk essentially meaningless.
It’s actually comical when some people say things like “We need not be concerned because the probability of human extinction due to AGI is SO LOW”
rod jenkin: your point ("don't make AGI") resonates with many thinkers' ideas that come forward here-and-there but seem to stay "out of focus" for those who have a seat on the bandwagon. I refer to this link from the Convivial Society where some in and out of the field of AI question the "myth of technological inevitability," and to a relatively long quote from that session below the link:
"There is no better way to reinforce the myth of technological inevitability than to stage the ubiquity of AI in such a way that it renders the adoption of AI a fait accompli. . . .I should acknowledge that while there is no inevitability, agency and responsibility are unequally distributed. Thus, it is worth noting that the strategy of manufacturing inevitability has the effect of obfuscating responsibility, especially on the part of those who in fact have the greatest agency over the shape of the techno-economic structures that order contemporary society for the rest of us.
"The pioneering computer scientist, Joseph Weizenbaum, told us as much nearly 50 years ago in Computer Power and Human Reason: 'The myth of technological and political and social inevitability is a powerful tranquilizer of the conscience. Its service is to remove responsibility from the shoulders of everyone who truly believes in it. But in fact there are actors.' . . . The myth of technological inevitability is a powerful tranquilizer of the conscience. It bears repeating. . . .More from Weizenbaum, who writes with refreshing conviction:
“'But just as I have no license to dictate the actions of others, neither do the constructors of the world in which I must live have a right to unconditionally impose their visions on me. Scientists and technologists have, because of their power, an especially heavy responsibility, one that is not to be sloughed off behind a facade of slogans such as that of technological inevitability.'
"But Weizenbaum understood one more thing of consequence: the necessity of courage. Allow me to quote him at length:
"'I recently heard an officer of a great university publicly defend an important policy decision he had made, one that many of the university’s students and faculty opposed on moral grounds, with the words: ‘We could have taken a moral stand, but what good would that have done?’ But the good of a moral act inheres in the act itself. That is why an act can itself ennoble or corrupt the person who performs it. The victory of instrumental reason in our time has brought about the virtual disappearance of this insight and thus perforce the delegitimation of the very idea of nobility.'
"'I am aware, of course, that hardly anyone who reads these lines will feel himself addressed by them—so deep has the conviction that we are all governed by anonymous forces beyond our control penetrated into the shared consciousness of our time.'
And accompanying this conviction is a debasement of the idea of civil courage.'
"'It is a widely held but a grievously mistaken belief that civil courage finds exercise only in the context of world-shaking events. To the contrary, its most arduous exercise is often in those small contexts in which the challenge is to overcome the fears induced by petty concerns over career, over our relationships to those who appear to have power over us, over whatever may disturb the tranquility of our mundane existence.'"
The writer of the blog writes: "While the degree of agency we share over the shape of our world varies greatly, I remain convinced that we all have choices to make. But these choices are not without consequences or costs. And each one of us will find, from time to time, the need for courage, and it strikes me that such courage, call it civil courage or courage in the ordinary, is the antidote to what Arendt famously diagnosed as the banality of evil."
Mehdididit: I am so glad to see this--and that list of signees is really encouraging. There are also a few serious "OR ELSE" factors "salted' throughout the document. This, again, according to differences in states, but not so much differences in what is needed and expected from these companies. Also, 60 MINUTES did a great service in their reporting of the problem--which I had not seen much about even though I keep up with things--so there still needs to be good and consistent coverage. It takes time for "the public" to get ahold of a problem this large.
I saw overnight on BBC that a family in Australia is suing META for damage caused to their daughter. (I missed a good part of it and hope they show it again.) (12-17)
Also, speaking of predictions, it's not like this problem or its expansiveness was NOT PREDICTABLE--if they don't grasp the potential problems just from having lived in a civilized culture (ahem) OR from having a good formal academic background (ahem again), or from just walking around in the world for over 30 years with other human beings, then they are either really unintelligent/stupid or in a state of ego-centric SERIOUS DENIAL and they need to get some personal help <--not a joke. In the general scheme of things, there is no difference between the Silicon-Valley titans and the oil, gas, and chemical people, who carelessly continue to find ways to foul the world and poison everyone in it.
What's also predictable (and what many have been screaming about for decades) is the penchant towards power-grabbing ("because I can"), which may carry methods of shielding, e.g. gating, bribing, fear, and extortion, but do not diminish a person's responsibility to patrol oneself in terms of some basic human principles. (Read about the Ring of Gyges in Plato's Republic.) It's not like we couldn't or did not know.
These people, however, share that they are involved in the abuse of freedoms, both on the part of the companies' people concerning their own freedoms (misunderstood as license), but also of their abuse of freedoms of young people and their families who feel there is an invisible monster in the house.
And the abuse of freedoms, in the end, calls down laws because, if we don't master ourselves, someone will think it important to master us and everyone else, and be right to do so. It's the chink in the armor of democracies--that self-control/mastery is the fuel that continues to ignite freedoms, as long as we can keep them.
I'm surprised at how little commentary there is on the data acquisition for AI vs human intelligence.
Human "data collection" is generally an ensemble of analog physical and chemical reactions that result in neurological and brain responses. When we are cold it's because trillions or at least millions cells in our body are undergoing temperature-related changes. That triggers physiological responses that feed our nervous system at numerous levels. Some of those changes result in sensations, others in emotions, and then together with a bunch of feedback loops end up in thoughts. It's analog, parallel at numerous physical and biological scales, and can be experienced in numerous ways, which differ among people even under similar physical conditions. That's our interface to reality, which itself is variable at all sorts of scales and in dimensions that often defy enumeration. That's a far cry from the digital representation of "cold" a collection of thermometers feeding signals to something and using classifiers and/or logic to try to establish what cold "is".
I'm neither neuroscientist nor an AI expert, but I am an alert human who has learned a from psychology, and it seems really clear that human cognition is profoundly related to experiences, emotions, and perceptions. Work on trauma shows that integrating experiences into memories and a stable sense of self involves connecting all these methods of interacting with reality.
I'd actually be surprised if our ability to generalize things into abstractions was NOT related to the synthesis that happens when we transform all those "inputs" into a "memory", a "sense of self", and on some level a "world model". Perhaps these human forms of synthesis allow us to deal stably and creatively with the many elements of reality that cannot be neatly instrumented, and still converge on truth in cases where objective reality is clear.
As far as I can tell, current and even novel AI approaches still basically trying to take a large corpus of digital data as a robust representation of reality. That will work for problem spaces that are already amenable to mathematical representations of reality, but I'd speculate that they will continue to struggle with other forms of abstraction or representations of reality.
Hi Gary! As always thanks for your clear words and thought experiments. They always make for engaging reading. I’m a neuroscientist by training (though I do old fashioned software product management with gen AI warily used as a companion by day) and my thoughts below will reflect that bias.
From an AI generated transcript of the interview (Using the Whisper apps on device model):
>I think it’s going to require a whole redesign where the world models are first-class citizens and the correlations are secondary rather than the other way around, where the correlations are kind of the main thing we got going on and people are trying to somehow jam a world model into them. I think the next generation of AI will be all about world models.
Would I be right in thinking of this as an architecture where there *is* a kind of homunculus, but one we (humans) program in and set the terms for its evolution based on downstream interactions where LLMs come in?
Two other thoughts, assuming this is a reasonable picture of the architecture:
1. This makes it very crucial that world models not be themselves overly biased in the wrong axes. They are serving as an uber-constitution for the overall AI, enforcing rules that shape the “physics” of the computation that follows.
I believe it is possible for humans to coordinate and build and update such rulesets for AI to then operate on. But it’s crucial to ensure this doesn’t become a new vector for control and manipulation.
2. This seems to me to be a mechanism for coordination. Where these models serve as a kind of intermediary substrate for human communication. But whether this enhances or stultifies communication will be, I think, up to the care with which we design the interfaces and transactional/translational surfaces. That is, we are now gamifying the human-computer interface, with the goal that the design frameworks that emerge best enhance human communication and coordination.
How to endow AI with true understanding has been the eternal quest.
Explicit representations and explicit models for all objects involved did not work out because these are very fuzzy things that can't be put in a schema.
The current approach with LLMs is mostly symbolic than anything. Each entity has a vast number of relationships with other entities, that partially defines its properties. The rules for how to manipulate these are based on examples, rather than symbolic logic, which, if anything, works better in practice.
When deeper knowledge is needed, the models now call various external means such as software, etc. This approach has been truly revolutionary, really.
LeCun and Fei-Fei Li are trying to do better. It seems it will still be neural nets that learn from experience, but the data will be more heavily skewed towards perception rather than language, and the representations will still be implicit.
Nobody came up with anything better. Clearly what is in our heads is a lot more holistic, but how to do that?
Professor Marcus, your call for 'world models' is vital. As a pedagogue i see striking parallels between how children (like those with hearing impairments) reconstruct a world from fragments and how AI struggles with 'understanding' vs. 'prediction'.
I am developing the R-Omega Protocol to address exactly this: moving from statistical prediction to a robust architecture that allows an AI to hesitate and even say 'No'—the ultimate sign of a world model with integrity.
I’ve documented these pedagogical transfers (e.g., 'The Silent Language') on my page. Thank you for championing depth over scale. www.project-robert.de
12-20-25 Below is a NYTimes gift article re: New York Governor signing a bill about regulating AI to better match California's latest regulation. I was wondering why Gavin Newsom's name was not on that signee area in the letter from States' AG association giving an account of the present and developing legal situation applied to AI safety and regarding regulations posted here earlier.
I though the oddest comment in the article was that the New York governor was "raising the bar" for AI safety by following California's example, which sounds good; but (it seems to me) raising the bar when there was no real bar in the first place is a rather telling play on words. (So, what else is new?) But quibble aside, perhaps she was referring to the honest concern and applications of it already coming from the big-tech industry?
Also, again, the article refers to the support for the NY governor of big tech companies noting that they have unleashed millions of $$ to lobbyists to do what they can to limit state regulations--this is after doing the same to enhance Trump's move to regulate the regulations, Trump style, for the people, of course.
BTW, BBC has an excellent report going (to be replayed over the weekend and on their website) about how bad actors are getting a leg-up on hacking into all sorts of institutions.
A wonderful insight into what a world model would provide. However it sounds like another form of cross-domain reasoning. Whether or not cross-domain reasoning truly is an understanding of the world is another matter. However, it does lend itself to stressing and finding constraints from one domain to one or more others without losing meaning or undergoing coherence collapse. Current frontier models are already amenable to this type of reasoning, however.
James: If I may, there are of course different and clearly distinct fields of study, i.e., domains, and philosophy is one of those domains.
However, all domains, including philosophy, also have philosophical foundations which rest on what is inborn and performative about conscious process and, as learned, is what everyone assumes not only about specific knowledge domains, but knowing and knowledge itself, which all domains stand on, and which is one of the data sets of philosophy itself.
As it happens in history, scientific method and its adapted principles and protocols lend system to what is commonly referred to as scientific/empirical method though, as applied, the kinds of data that one studies in any specific field can import greatly on what one takes as evidence and so on what sub-methods one uses to achieve and share it, and though the general idea of rules of evidence hold.
And so, what you refer to as "cross-domain reasoning" of course must take into consideration different contents using control of comparative/contrasting principles and, as with any data, one's sub-methods are nuanced accordingly.
However, reasoning as object content of one's desire to understand, regardless of specified content (or by the protocols of science), and reasoning that one automatically uses in the performance of thinking to come to a critical understanding of any x, DIFFER . . . but only insofar as spontaneous performance, though developmental, comes with being human, while the other is learned and systematized and so, as learned-about, can be in error.
Insofar as knowledge is assumed to depict the reality of the data under consideration, the aim of reasoning, whether inborn or learned, is tacitly unified in the scientist, and it becomes the basis of all successful collaboration and critique.
Scientific method, however, is a learnable way to reason in critical and replicative fashion about WHAT x is; and if that reasoning works (we begin to understand X), which most assume is a matter of collecting and verifying concrete evidence and nuances of that x order, then it has its roots in the inborn nature of consciousness and so of how all minds work as the basis of human living, pointing to a sort of "trans-cultural base" as an element in philosophical understanding--of reasoning, regardless of content.
If I am correct in this matter then, and insofar as one's learned method of understanding is in error or lacking in some way, whether in cross-field studies or not, then the idea of its being "in error" stands or falls according to what is inborn and performative in us all and whether we know and apply it rightly.
C. King: If I may clarify my position. I agree that in the course of human learning, we may do this via experimentation/scientific method. We also learn by reading the experiences of others. Whether or not one method is more effective than the other is less of an issue, but rather than did new understanding occur with either method. If so, why? Our world (from the materialist standpoint) is encoded in our neural system in which we carry immutable coherent domains in multiple disciplines. For example, if we learn by scientific experimentation method, we make an observation of the phenomenon of interest, we characterize it against adjacent coherent domains, such as mathematics, other scientific disciplines, informational, intuitive (does it make sense), philosophical (where to classify) domains. We take those priors and test them against the new observation. We make sure it is consistent and fit into our world view framework. We internally adjust the coherence bands, identifying constraints in order to prevent semantic bleeding (loss of information or meaning of a system) and over-dominance of the new data (coherence collapse, over repetition and enforcement). That we also are able to learn from written literature (or spoken, observation, for that matter) and do a similar manipulation internally means that understanding the world that we inhabit without direct experience. Where the literature might tell us the mentioned constraints, many times we need to adjust it into our world view internally. Regarding an AI, it does not have the benefit of direct observation, other than interacting with the prompter. It's pre- and post- training material is all it has to establish those priors. Its world is from our literature. All tokens are flat and no real meaning (understanding) among the various coherent domains was given. The post-training RL H/AI F is an attempt at giving a topography to this LLM knowledge landscape. But even within our literature, there are "meta-literatures," which was never given elevated meaning. Science examples might be David Bohm's "Wholeness and the Implicate Order" and Ilya Prigogine's work on the Brusselator (self-organizing systems). These literatures, for example, cross multiple domains -- informatics, science, metaphysics, experiential/phenomenological. As such the data needs to exercise adjusting the appropriate coherence bands to incorporate it into understanding. But during training, these texts are flat and do not lend itself to a higher meaning. But the GPT5 class frontier model seems to recognize the situation and is capable of reorganizing literatures around a set of priors if prompted to do so. GPT4o was unable to do this.
James: Your response is helpful. It is in part what I wanted to do by signing onto this site--to get a sense of what's going on in the "training" etc., and particularly the philosophical assumptions operating in the field of A.I.
YOU WRITE: "Our world (from the materialist standpoint) is encoded in our neural system in which we carry immutable coherent domains in multiple disciplines. For example, if we learn by scientific experimentation method, we make an observation of the phenomenon of interest, we characterize it against adjacent coherent domains, such as mathematics, other scientific disciplines, . ."
I think the "bleeding" metaphor is (probably, depending on definitions) a part of the above paragraph as the question "what about the materialist standpoint accounts for learning as such?" I'm not sure, but it sounds like a confrontation model of cognition is hidden in the philosophical assumptions active in the above text? (There's nothing uncommon about that, however, for sure, in pretty-much all of the sciences to date.)
But we are way beyond what can be accomplished on a blog. Thank you for your explanation--very helpful.
Wow, 5 ads in 11 minutes. I can't wait for those ads to be dynamically LLM-generated and be referencing me by name and how much milk I have in my fridge. /s
Anyway Gary I appreciate your work in continuing to highlight LLM absurdity even at the cost of your professional reputation. However it's pretty obvious to understand why world models are important, but I have to ask why should we build AGI at all? That always seems to be assumed by all the media and posts about world models. But the thought experiments/logical conclusions of what would.happen if there's infinite AI agents doing crap on the Internet and the real world is bad, and it's just as bad if those agents were actually capable of anything.
Would appreciate if you could cover this topic in a post at some point. What's your case for we should build AGI?
Blue Owl proved it isn't entirely stupid. Cancel on the Michigan data center financing. $10B - oddly the same amount the press was touting yesterday that Altman was trying to coax out of Amazon.
Gary,
Thank you for citing and elevating a trusted source of business news that also always manages to humanize business stories, business data, and the ever perplexing gyrations of the economy. Long time fan of Markeplace, Kai and his team and his process. Enjoy your SubStack and its focus. Making ongoing sense of the rapidly evolving landscape with AI infiltrating as much as it is allowed together away with…can be exhausting and perplexing.
Warmly,
Deborah and the Credtent team
"understand, not just predict'
"bar" has three parts of speech, as a noun it has 20 meanings - what good is prediction by an LLM if it doesn't knoiw the meaning? The statement has the wrong basis.
And don’t forget LLMs predict tokens, not words, so here come (imagined) barbaric, fubar, foobar, barking, barber, rhubarb, abarth, babar, barrel, ….
Bringing GOFAI back in certain ways is good, but ultimately, we're stuck with arbitration instead of any sort of real "understanding" of any kind.
A very rough example of pseudocode to illustrate this arbitrary relationship:
let p="night"
input R
if R="day" then print p+"is"+R
Now, if I type "day", then the output would be "night is day". Great. Absolutely "correct output" according to its programming. It doesn’t necessarily "make sense" but it doesn’t have to because it’s the programming! The same goes with any other input that gets fed into the machine to produce output e.g., "nLc is auS", "e8jey is 3uD4", and so on.
To the machine, codes and inputs are nothing more than items and sequences to execute. There’s no meaning to this sequencing or execution activity to the machine. To the programmer, there is meaning because he or she conceptualizes and understands variables as representative placeholders of their conscious experiences. The machine doesn’t comprehend concepts such as "variables", "placeholders", "items", "sequences", "execution", etc. It just doesn’t comprehend, period. Thus, a machine never truly "knows" what it’s doing and can only take on the operational appearance of comprehension.
Due to what machines are, machines are stuck with pretend-representation and pretend-understanding. Machines are epistemically land-locked. There's no externality to them, they don't possess, and can't possess, intentionality (cf. term in Philosophy of Mind https://plato.stanford.edu/entries/intentionality/ ) as such, there's no such thing as actual grounding in machines.
I guess the (IMHO) false hope of current gen AI is that “knowing” pops up as an emergent property of a sufficiently large neural network. The hope is based on similarity between neurons in our brains and nodes in the artificial neural network. Individual neurons also do not “know”, neither do the proteins they are built from or metabolic processes that power the whole thing. Yet the brain (or the human organism as a whole) seems to “know”. Why I think the hope of getting this emergent property in a neural network is false? Because any neural network is a static non-linear mapping from inputs to outputs. “Knowing” (or if you will consciousness) requires temporal evolution and I believe also continuous learning and embodied existence (possibly also self reproduction to give it motivation to “do” anything). That seems to be, as of now, out of reach for our silicon based technology and synchronous state automatons that all current computers are. It may be out of reach of digital technology in general. Frankly, I hope it remains that way ;-).
One of the reasons I don’t expect any useful emergent properties from neural nets is that they’re really not at all like neurons in their functioning or evolution over time.
I doubt it will remain inaccessible to silicon forever. These things have “emerged” and been played with over evolution multiple times. I see no fundamental reason they need to be fully substrate dependent.
I think it’s fair to say silicon won’t naturally evolve this kind of intelligence, at least on the surface of the Earth.
But that’s a far cry from what biology as designer and tool user can achieve.
I’ve recently read a study making an argument that intelligence may not be computable. I unfortunately cannot find the link anymore :-(. It is possible that intelligence similar to human is not possible in a digital system due to the necessarily incomplete representation of reality caused by digitization. It still may be possible in an analog system. The other claim was that self-reproduction may be needed for any agency and consciousness.
I’d agree with that depending on what they define intelligence to be.
But I don’t know why intelligence similar to a human should be the goal at all? At least so far, no one has assumed the best way to get humans far is to build faster legs to attach to our hips.
Why should we build intelligence in a similarly bizarre fashion? That’s where I hope *computing* goes. Whether we end up calling it AI or whatever else.
Yes, I also don’t understand why AGI (pretty much regardless of definition) should by a worthy goal to pursue. I think we need technology to augment humans, not to emulate them. I don’t want any AI system to have independent opinion and goals.
If you happen to remember the title or author of the article I'd love to read it. I have been thinking about similar things and made a comment to that effect below - would love to learn what scholars of the subject are saying.
I’ve found the link: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1362658/full
Thank you!
Individual neurons also do not “know”
I’m not so sure,
Neurons (and all other cells) do a truly massive amount of information processing about which we have only the most rudimentary knowledge.
Immune cells can learn to recognize and then kill viruses and bacteria.
While some might say this is really not “knowing”, how do they know that?
Lots of uncertainty but one thing is pretty clear: treating neurons as simple “weights” in a neural net is a gross oversimplification (at best) and just plain wrong (at worst).
You’re shifting goal posts, and in the process twisting my argument.
Again:
Life has agency
Sub parts of life have agency, going all the way down to where it’s just lifeless molecules. That is, the whole has a greater agency than the sum of its parts, but that doesn’t take away that the parts have agency.
Individual cells, especially do have agency. We also shape the fate of individual cells, but by “nature” and by choices we make, dietary, exposure, etc.
Individual cells can now be fitted with additional optogenetic proteins at the cell surface.
This means we can play with and control their agency.
We are also well on the path to designing other proteins and being able to integrate them into cells.
We can thus add design to life and control it.
We can design non life and control it.
Sub parts of our own body can and do go rogue and respond to signals “sub”-consciously , integrating signals that we consciously discount.
Either we’re a sum of agents who can go rogue, and this obviously changes what “life” or “self” means.
Or we can design agency that expands our own.
It’s a fine distinction but either way your hard line is obliterated by evidence. Machines can be designed to mimic life. Thus they can have agency the same way sub parts of life can have it.
Can they have equal agency to a human who is a sum of so many distinct cellular agents (more bacteria in the gut than your entire rest of the body… so you and I aren’t even genomically one thing)?
I doubt anytime soon. And I doubt we should want to do it. But I won’t say with certainty we *cannot* do it. Right now that’s an article of faith. Not factual science. Insisting it is means some other idiot can proceed with a design that screws me, and I’m a selfish piece of biology. I’d rather be on the drivers seat, not be the car.
I think Gary's point in this interview has been world models, rather than symbolic logic. So, first the words are given meaning, and only then any transformations are attempted.
"World models" translates to "what's programmed in." That's arbitrated. "Given meaning" is a hand-wave. It doesn't happen just because arbitration or matching happens.
I propose for now we defer the issue of how world models are implemented. The interview was very vague and focused on high level discussion of how to give machines world models and representations, including attempts by LeCun and Fei-Fei Li.
Of course there is bitter disagreement for how such "meaning" is to be put in, and if it is even possible at all.
It's a start, as we all know. However, LeCun as well as everyone else are still avoiding the "The Hard Problem of Machine Understanding" which is grounding. Somebody has to get to it... Otherwise every machine is going to be stuck at the behaviorist stage.
Is the “hard problem of machine understanding” a formal term? Seems to me it’s a restatement of the hard problem of consciousness, for machines, from the sound of things.
The hard problem of consciousness don't really have much to do with practical matters, while grounding is a technical issue that has to be eventually faked well enough to handle any task. Grounding doesn't have to be taken care of perfectly, just well enough that performance issues disappear. For example, satellites in orbit doesn't take into account many relativistic effects, yet still operate well enough so the omissions don't matter.
That is fair enough.
How to do grounding is a very long discussion. I happen to think that we have made very good progress with approximate means, with more to come, though I know folks as yourself will strenuously disagree, arguing it is dumb symbols all the way down.
As long as the performance eventually gets faked well enough, the theoretics can take a back seat. The trouble is all sorts of issues are rearing their ugly heads in practice, so "well enough" still isn't anywhere in sight.
Perhaps I don’t. What’s the operating definition of design you’re using in this discussion?
Is it possible to agree with you that AGI is going to be hard, new architecture is needed and scaling is not all we need...
...and yet worry that Eliezer is right that if we do finally create AGI / ASI, that this will be our last invention.
Unless you can refute orthogonality / gradient descent and show how ASI could possibly be safe shouldnt you be saying "Don't make AGI" as your primary statement rather than telling people they are going about this wrong?
Amen. There should be no discussion of how to build AGI before there is overwhelming consensus agreement among all stakeholders (i.e., the entire human species) that AGI ought to be built in the first place. Me? I say "no."
There is a weird combination of faith, hubris and in some cases megalomania involved in the assumption that pursuing ASI is the proper goal for humanity.
The AI field is more a religion than a science. The practitioners behave like high priests (even Demi-gods), as if they are uniquely qualified and sanctioned to decide, speak and act for the rest of humanity.
And Then It Fell: Of the many branches of possible uses of broader AGI (as suggested in Gary's note) the worst is to imagine it in the hands of those who get off on surveillance, for any means or ends.
As I understand it, even if goods can come from it, and I'm sure someone will find something somewhere, it doesn't matter precisely because AGI is an article of intelligence; and insofar as raw intelligence, under the fuller application of human agency, can opt for the full spectrum of good and bad, articles of intelligence can be developed or accessed by an intelligent person acting under the influence of any part of that spectrum for any situational disturbance, for accidental or deliberate scenarios, and/or under conscious, partly conscious, or unconscious motivations.
. . . and we don't want to attack human agency as such, or the freedom side of the freedom-to-responsibility complex, or do we? And what would it mean to be a noble person, even if one could, if everyone else is under one's watchful eye or already dead? I could go on, but you probably get the picture.
Probably the only thing CERTAIN about the trajectory of AI is that it will take on an ever more (probably completely) encompassing surveillance role.
Even current AI capabilities would have brought members of the East German Stasi to orgasm. They would have spent every waking hour in the AI-gasmatron.
Larry Jewett: Nothing like a fitting set of metaphors.
Metaphors?
Ha.
I was being quite literal.
After all, the Stasi are known to have gotten their jollies with bedroom surveillance.
The mistake that many a proponent of AGI/ASI makes is assuming that risk depends only on the probability of an event.
The reality (as every insurance company knows) is that
Risk = probability X IMPACT
So even (assumed) low probability high impact events (eg, end of democracy, societal breakdown or even human extinction) can have high associated risk.(what “impact/cost” should we assign to human extinction, by the way? )
Many proponents of AGI try to dismiss concerns by stating that human extinction is highly unlikely (whatever “highly” means). While that is probably (albeit by no means certainly) the case, extinction is hardly the only possible high impact event (or even the one people should necessarily focus on)
And simply considering probability (which no one really has anything but the foggiest notion of, anyway) without factoring in impact renders all talk of AGI risk essentially meaningless.
It’s actually comical when some people say things like “We need not be concerned because the probability of human extinction due to AGI is SO LOW”
rod jenkin: your point ("don't make AGI") resonates with many thinkers' ideas that come forward here-and-there but seem to stay "out of focus" for those who have a seat on the bandwagon. I refer to this link from the Convivial Society where some in and out of the field of AI question the "myth of technological inevitability," and to a relatively long quote from that session below the link:
https://open.substack.com/pub/theconvivialsociety/p/manufactured-inevitability-and-the?utm_campaign=post&utm_medium=email
"There is no better way to reinforce the myth of technological inevitability than to stage the ubiquity of AI in such a way that it renders the adoption of AI a fait accompli. . . .I should acknowledge that while there is no inevitability, agency and responsibility are unequally distributed. Thus, it is worth noting that the strategy of manufacturing inevitability has the effect of obfuscating responsibility, especially on the part of those who in fact have the greatest agency over the shape of the techno-economic structures that order contemporary society for the rest of us.
"The pioneering computer scientist, Joseph Weizenbaum, told us as much nearly 50 years ago in Computer Power and Human Reason: 'The myth of technological and political and social inevitability is a powerful tranquilizer of the conscience. Its service is to remove responsibility from the shoulders of everyone who truly believes in it. But in fact there are actors.' . . . The myth of technological inevitability is a powerful tranquilizer of the conscience. It bears repeating. . . .More from Weizenbaum, who writes with refreshing conviction:
“'But just as I have no license to dictate the actions of others, neither do the constructors of the world in which I must live have a right to unconditionally impose their visions on me. Scientists and technologists have, because of their power, an especially heavy responsibility, one that is not to be sloughed off behind a facade of slogans such as that of technological inevitability.'
"But Weizenbaum understood one more thing of consequence: the necessity of courage. Allow me to quote him at length:
"'I recently heard an officer of a great university publicly defend an important policy decision he had made, one that many of the university’s students and faculty opposed on moral grounds, with the words: ‘We could have taken a moral stand, but what good would that have done?’ But the good of a moral act inheres in the act itself. That is why an act can itself ennoble or corrupt the person who performs it. The victory of instrumental reason in our time has brought about the virtual disappearance of this insight and thus perforce the delegitimation of the very idea of nobility.'
"'I am aware, of course, that hardly anyone who reads these lines will feel himself addressed by them—so deep has the conviction that we are all governed by anonymous forces beyond our control penetrated into the shared consciousness of our time.'
And accompanying this conviction is a debasement of the idea of civil courage.'
"'It is a widely held but a grievously mistaken belief that civil courage finds exercise only in the context of world-shaking events. To the contrary, its most arduous exercise is often in those small contexts in which the challenge is to overcome the fears induced by petty concerns over career, over our relationships to those who appear to have power over us, over whatever may disturb the tranquility of our mundane existence.'"
The writer of the blog writes: "While the degree of agency we share over the shape of our world varies greatly, I remain convinced that we all have choices to make. But these choices are not without consequences or costs. And each one of us will find, from time to time, the need for courage, and it strikes me that such courage, call it civil courage or courage in the ordinary, is the antidote to what Arendt famously diagnosed as the banality of evil."
Here is Gary on CNN:
https://www.msn.com/en-us/news/politics/ai-regulation-or-the-wild-west/vi-AA1ShYYc?ocid=msedgdhp&pc=HCTS&cvid=6942365b077849678f28ce0007252e3c&ei=23
Related, on state regulation of AI
https://www.njoag.gov/wp-content/uploads/2025/12/2025-1210_AI-Multistate-Letter.pdf
It essentially says that most states already have laws on the books that govern AI. It went out 2 days before the EO.
Mehdididit: I am so glad to see this--and that list of signees is really encouraging. There are also a few serious "OR ELSE" factors "salted' throughout the document. This, again, according to differences in states, but not so much differences in what is needed and expected from these companies. Also, 60 MINUTES did a great service in their reporting of the problem--which I had not seen much about even though I keep up with things--so there still needs to be good and consistent coverage. It takes time for "the public" to get ahold of a problem this large.
I saw overnight on BBC that a family in Australia is suing META for damage caused to their daughter. (I missed a good part of it and hope they show it again.) (12-17)
Also, speaking of predictions, it's not like this problem or its expansiveness was NOT PREDICTABLE--if they don't grasp the potential problems just from having lived in a civilized culture (ahem) OR from having a good formal academic background (ahem again), or from just walking around in the world for over 30 years with other human beings, then they are either really unintelligent/stupid or in a state of ego-centric SERIOUS DENIAL and they need to get some personal help <--not a joke. In the general scheme of things, there is no difference between the Silicon-Valley titans and the oil, gas, and chemical people, who carelessly continue to find ways to foul the world and poison everyone in it.
What's also predictable (and what many have been screaming about for decades) is the penchant towards power-grabbing ("because I can"), which may carry methods of shielding, e.g. gating, bribing, fear, and extortion, but do not diminish a person's responsibility to patrol oneself in terms of some basic human principles. (Read about the Ring of Gyges in Plato's Republic.) It's not like we couldn't or did not know.
These people, however, share that they are involved in the abuse of freedoms, both on the part of the companies' people concerning their own freedoms (misunderstood as license), but also of their abuse of freedoms of young people and their families who feel there is an invisible monster in the house.
And the abuse of freedoms, in the end, calls down laws because, if we don't master ourselves, someone will think it important to master us and everyone else, and be right to do so. It's the chink in the armor of democracies--that self-control/mastery is the fuel that continues to ignite freedoms, as long as we can keep them.
Gary, what's preventing at least a solid demonstration of world models as the primary driver
I'm surprised at how little commentary there is on the data acquisition for AI vs human intelligence.
Human "data collection" is generally an ensemble of analog physical and chemical reactions that result in neurological and brain responses. When we are cold it's because trillions or at least millions cells in our body are undergoing temperature-related changes. That triggers physiological responses that feed our nervous system at numerous levels. Some of those changes result in sensations, others in emotions, and then together with a bunch of feedback loops end up in thoughts. It's analog, parallel at numerous physical and biological scales, and can be experienced in numerous ways, which differ among people even under similar physical conditions. That's our interface to reality, which itself is variable at all sorts of scales and in dimensions that often defy enumeration. That's a far cry from the digital representation of "cold" a collection of thermometers feeding signals to something and using classifiers and/or logic to try to establish what cold "is".
I'm neither neuroscientist nor an AI expert, but I am an alert human who has learned a from psychology, and it seems really clear that human cognition is profoundly related to experiences, emotions, and perceptions. Work on trauma shows that integrating experiences into memories and a stable sense of self involves connecting all these methods of interacting with reality.
I'd actually be surprised if our ability to generalize things into abstractions was NOT related to the synthesis that happens when we transform all those "inputs" into a "memory", a "sense of self", and on some level a "world model". Perhaps these human forms of synthesis allow us to deal stably and creatively with the many elements of reality that cannot be neatly instrumented, and still converge on truth in cases where objective reality is clear.
As far as I can tell, current and even novel AI approaches still basically trying to take a large corpus of digital data as a robust representation of reality. That will work for problem spaces that are already amenable to mathematical representations of reality, but I'd speculate that they will continue to struggle with other forms of abstraction or representations of reality.
Hi Gary! As always thanks for your clear words and thought experiments. They always make for engaging reading. I’m a neuroscientist by training (though I do old fashioned software product management with gen AI warily used as a companion by day) and my thoughts below will reflect that bias.
From an AI generated transcript of the interview (Using the Whisper apps on device model):
>I think it’s going to require a whole redesign where the world models are first-class citizens and the correlations are secondary rather than the other way around, where the correlations are kind of the main thing we got going on and people are trying to somehow jam a world model into them. I think the next generation of AI will be all about world models.
Would I be right in thinking of this as an architecture where there *is* a kind of homunculus, but one we (humans) program in and set the terms for its evolution based on downstream interactions where LLMs come in?
Two other thoughts, assuming this is a reasonable picture of the architecture:
1. This makes it very crucial that world models not be themselves overly biased in the wrong axes. They are serving as an uber-constitution for the overall AI, enforcing rules that shape the “physics” of the computation that follows.
I believe it is possible for humans to coordinate and build and update such rulesets for AI to then operate on. But it’s crucial to ensure this doesn’t become a new vector for control and manipulation.
2. This seems to me to be a mechanism for coordination. Where these models serve as a kind of intermediary substrate for human communication. But whether this enhances or stultifies communication will be, I think, up to the care with which we design the interfaces and transactional/translational surfaces. That is, we are now gamifying the human-computer interface, with the goal that the design frameworks that emerge best enhance human communication and coordination.
How to endow AI with true understanding has been the eternal quest.
Explicit representations and explicit models for all objects involved did not work out because these are very fuzzy things that can't be put in a schema.
The current approach with LLMs is mostly symbolic than anything. Each entity has a vast number of relationships with other entities, that partially defines its properties. The rules for how to manipulate these are based on examples, rather than symbolic logic, which, if anything, works better in practice.
When deeper knowledge is needed, the models now call various external means such as software, etc. This approach has been truly revolutionary, really.
LeCun and Fei-Fei Li are trying to do better. It seems it will still be neural nets that learn from experience, but the data will be more heavily skewed towards perception rather than language, and the representations will still be implicit.
Nobody came up with anything better. Clearly what is in our heads is a lot more holistic, but how to do that?
Professor Marcus, your call for 'world models' is vital. As a pedagogue i see striking parallels between how children (like those with hearing impairments) reconstruct a world from fragments and how AI struggles with 'understanding' vs. 'prediction'.
I am developing the R-Omega Protocol to address exactly this: moving from statistical prediction to a robust architecture that allows an AI to hesitate and even say 'No'—the ultimate sign of a world model with integrity.
I’ve documented these pedagogical transfers (e.g., 'The Silent Language') on my page. Thank you for championing depth over scale. www.project-robert.de
12-20-25 Below is a NYTimes gift article re: New York Governor signing a bill about regulating AI to better match California's latest regulation. I was wondering why Gavin Newsom's name was not on that signee area in the letter from States' AG association giving an account of the present and developing legal situation applied to AI safety and regarding regulations posted here earlier.
I though the oddest comment in the article was that the New York governor was "raising the bar" for AI safety by following California's example, which sounds good; but (it seems to me) raising the bar when there was no real bar in the first place is a rather telling play on words. (So, what else is new?) But quibble aside, perhaps she was referring to the honest concern and applications of it already coming from the big-tech industry?
Also, again, the article refers to the support for the NY governor of big tech companies noting that they have unleashed millions of $$ to lobbyists to do what they can to limit state regulations--this is after doing the same to enhance Trump's move to regulate the regulations, Trump style, for the people, of course.
BTW, BBC has an excellent report going (to be replayed over the weekend and on their website) about how bad actors are getting a leg-up on hacking into all sorts of institutions.
https://www.nytimes.com/2025/12/19/nyregion/ai-bill-regulations-ny.html?unlocked_article_code=1.-E8.hKKo.9sf13g_gMbFA&smid=url-share
A wonderful insight into what a world model would provide. However it sounds like another form of cross-domain reasoning. Whether or not cross-domain reasoning truly is an understanding of the world is another matter. However, it does lend itself to stressing and finding constraints from one domain to one or more others without losing meaning or undergoing coherence collapse. Current frontier models are already amenable to this type of reasoning, however.
James: If I may, there are of course different and clearly distinct fields of study, i.e., domains, and philosophy is one of those domains.
However, all domains, including philosophy, also have philosophical foundations which rest on what is inborn and performative about conscious process and, as learned, is what everyone assumes not only about specific knowledge domains, but knowing and knowledge itself, which all domains stand on, and which is one of the data sets of philosophy itself.
As it happens in history, scientific method and its adapted principles and protocols lend system to what is commonly referred to as scientific/empirical method though, as applied, the kinds of data that one studies in any specific field can import greatly on what one takes as evidence and so on what sub-methods one uses to achieve and share it, and though the general idea of rules of evidence hold.
And so, what you refer to as "cross-domain reasoning" of course must take into consideration different contents using control of comparative/contrasting principles and, as with any data, one's sub-methods are nuanced accordingly.
However, reasoning as object content of one's desire to understand, regardless of specified content (or by the protocols of science), and reasoning that one automatically uses in the performance of thinking to come to a critical understanding of any x, DIFFER . . . but only insofar as spontaneous performance, though developmental, comes with being human, while the other is learned and systematized and so, as learned-about, can be in error.
Insofar as knowledge is assumed to depict the reality of the data under consideration, the aim of reasoning, whether inborn or learned, is tacitly unified in the scientist, and it becomes the basis of all successful collaboration and critique.
Scientific method, however, is a learnable way to reason in critical and replicative fashion about WHAT x is; and if that reasoning works (we begin to understand X), which most assume is a matter of collecting and verifying concrete evidence and nuances of that x order, then it has its roots in the inborn nature of consciousness and so of how all minds work as the basis of human living, pointing to a sort of "trans-cultural base" as an element in philosophical understanding--of reasoning, regardless of content.
If I am correct in this matter then, and insofar as one's learned method of understanding is in error or lacking in some way, whether in cross-field studies or not, then the idea of its being "in error" stands or falls according to what is inborn and performative in us all and whether we know and apply it rightly.
C. King: If I may clarify my position. I agree that in the course of human learning, we may do this via experimentation/scientific method. We also learn by reading the experiences of others. Whether or not one method is more effective than the other is less of an issue, but rather than did new understanding occur with either method. If so, why? Our world (from the materialist standpoint) is encoded in our neural system in which we carry immutable coherent domains in multiple disciplines. For example, if we learn by scientific experimentation method, we make an observation of the phenomenon of interest, we characterize it against adjacent coherent domains, such as mathematics, other scientific disciplines, informational, intuitive (does it make sense), philosophical (where to classify) domains. We take those priors and test them against the new observation. We make sure it is consistent and fit into our world view framework. We internally adjust the coherence bands, identifying constraints in order to prevent semantic bleeding (loss of information or meaning of a system) and over-dominance of the new data (coherence collapse, over repetition and enforcement). That we also are able to learn from written literature (or spoken, observation, for that matter) and do a similar manipulation internally means that understanding the world that we inhabit without direct experience. Where the literature might tell us the mentioned constraints, many times we need to adjust it into our world view internally. Regarding an AI, it does not have the benefit of direct observation, other than interacting with the prompter. It's pre- and post- training material is all it has to establish those priors. Its world is from our literature. All tokens are flat and no real meaning (understanding) among the various coherent domains was given. The post-training RL H/AI F is an attempt at giving a topography to this LLM knowledge landscape. But even within our literature, there are "meta-literatures," which was never given elevated meaning. Science examples might be David Bohm's "Wholeness and the Implicate Order" and Ilya Prigogine's work on the Brusselator (self-organizing systems). These literatures, for example, cross multiple domains -- informatics, science, metaphysics, experiential/phenomenological. As such the data needs to exercise adjusting the appropriate coherence bands to incorporate it into understanding. But during training, these texts are flat and do not lend itself to a higher meaning. But the GPT5 class frontier model seems to recognize the situation and is capable of reorganizing literatures around a set of priors if prompted to do so. GPT4o was unable to do this.
James: Your response is helpful. It is in part what I wanted to do by signing onto this site--to get a sense of what's going on in the "training" etc., and particularly the philosophical assumptions operating in the field of A.I.
YOU WRITE: "Our world (from the materialist standpoint) is encoded in our neural system in which we carry immutable coherent domains in multiple disciplines. For example, if we learn by scientific experimentation method, we make an observation of the phenomenon of interest, we characterize it against adjacent coherent domains, such as mathematics, other scientific disciplines, . ."
I think the "bleeding" metaphor is (probably, depending on definitions) a part of the above paragraph as the question "what about the materialist standpoint accounts for learning as such?" I'm not sure, but it sounds like a confrontation model of cognition is hidden in the philosophical assumptions active in the above text? (There's nothing uncommon about that, however, for sure, in pretty-much all of the sciences to date.)
But we are way beyond what can be accomplished on a blog. Thank you for your explanation--very helpful.
Wow, 5 ads in 11 minutes. I can't wait for those ads to be dynamically LLM-generated and be referencing me by name and how much milk I have in my fridge. /s
Anyway Gary I appreciate your work in continuing to highlight LLM absurdity even at the cost of your professional reputation. However it's pretty obvious to understand why world models are important, but I have to ask why should we build AGI at all? That always seems to be assumed by all the media and posts about world models. But the thought experiments/logical conclusions of what would.happen if there's infinite AI agents doing crap on the Internet and the real world is bad, and it's just as bad if those agents were actually capable of anything.
Would appreciate if you could cover this topic in a post at some point. What's your case for we should build AGI?
Blue Owl proved it isn't entirely stupid. Cancel on the Michigan data center financing. $10B - oddly the same amount the press was touting yesterday that Altman was trying to coax out of Amazon.
When it comes to general intelligence (GI) understanding is necessary but not sufficient.
After all, lots of humans (with GI) have understanding but lack wisdom.
But if an understanding GI is hard to achieve, a wise GI (wise guy*) is even harder to come by.
*or more likely, “wise GAL” (General artificial Luminary)
Didn’t you hear?
The only thing holding back the appearance of AGI is human typing speed.