The review, as noted, indeed cuts off at, “And the most important thing we can…”
But it is Immediately followed by, “Enjoy unlimited digital access, free for 1 month”
That IS your guest link to read the full review. The publisher gets your email address, but zero cash from you, in exchange for a month of free reading (of this and other articles). It’s what publishers do to get visibility and some revenue (ultimately) to cover their costs for lending their support for, among others, Dr Marcus’ views.
What a silly thread to branch off from a serious topic.
I haven't read anything yet. But my first guess is that no one can deny the many goods that are already flowing from AI; on the other hand, the problems are also rife, it's hype is over the top, and (my guess also is) on that score, things are as Marcus has suggested.
Three people have tried to archive it and hit the same ellipses. Clearly we have to wait for a subscriber to archive the full article. I will keep checking.
If you think we should consider your arguments on this topic and "Don't dismiss" them, it seems an odd choice to put them behind a paywall, especially if the "pay" in question goes to News Corp, meaning (I believe) Rupert and Lachlan Murdoch. "A lot of people think" that these two are much more of a clear and immediate menace to humanity than any AGI/ASI is likely to be.
1. What makes a prediction useful? Roughly speaking: when it is verifiable, calibrated, based on the best available knowledge, and clear on the causal relationships.
2. The following is _not_ a conclusive way to evaluate an argument: when one argument merely reminds you of another argument you don’t like. The problem here is that you can cherry pick a comparison to help you torpedo an argument you don’t like. You can’t stop early when you find the argument you like! You have to be comprehensive. You have to fight the tendency to only search for support for your position.
3. Paying attention to the logic of an argument and evidence available is useful of course.
4. For a good Bayesian, the "outside view” might suggest… If all you know about an argument is that it predicts the world is going to end, you are sensible in assigning a low prior probability. But that’s just one part! As you consider more evidence, you must update your posterior probability accordingly.
5. By virtue of us being here (survivorship bias in a way), we know this world hasn’t ended yet! In other plausible counterfactuals, nuclear war could have broken out, followed by a nuclear winter.
6. Just a reminder about how predictions work. If in 1961 someone predicts a 20% chance of global nuclear war between 1961 and 1971, it doesn’t make them wrong if there is no such event happens.
Over the last several hundred years, there have been various occasions when some clever engineer created an artifact that seemed, at first, to the general public, to be alive. Toys that moved under their own power. Cuckoo clocks. Etc. In all cases, people eventually got over their shock and fear and realized that no, it's not alive, it's just a mechanism.
Here we have an especially complex, unpredictable, and powerful mechanism. Naturally, some people think it's a new life form. It's no longer part of the culture for them to express their fears in religious terms — that it is the work of Satan, driven by a demonic force — but they're saying essentially the same thing, for essentially the same reason: that properties they had previously taken to be exclusive to living beings were now being displayed by artifacts.
LLMs are no more being driven by demonic forces than cuckoo clocks were, but it is going to take some people a while to realize this.
So no, AI is not ever going to "go rogue". It is certainly possible that bad actors will use it in destructive ways; I'm not saying there's no cause for concern at all. But it's not going to wake up one day and decide, on its own, to kill us.
You are arguing your conclusion based on a claimed connection to a historical pattern. But this connection does not apply here, so your conclusion doesn't follow. You are arguing a strawman! Yudkowsky and Soares do not rely on "demonic forces" or any such foolishness.
Of course, they don't use any terms like "demonic". But they say things like "sufficiently smart AIs will develop goals of their own that put them in conflict with us". This concern is of the same kind as worries about demonic forces. How, exactly, will AIs develop goals of their own? They are machines. They do what they are built to do. I haven't heard Y&S even attempt to justify their claim; AFAIK, they take it as obvious, beyond the need to discuss. They're making the same mistake as those worried about the demonic forces driving cuckoo clocks: imputing animate qualities to the inanimate.
Granted, LLMs and other AI programs are fantastically complex artifacts whose behavior is beyond our ability to understand and predict; and this complexity makes it much easier to project sentience onto them. But it's still a projection.
I think we will have an AI apocalypse driven by bad human actors before we have one carried out by the AI acting on their own accord (whatever that means). Imagine robots performant and cheap enough to replace 50% of human physical labor in the next two decades. Or even sooner. We are not prepared…we have not even begun preparing for such a transition. So if there is anything left after that happens, then we’ll worry about the robots going on strike.
Can you please explain your logic in detail with a blog post somewhere? (Preferably someplace far away from Gary Marcus; he is a shining example of motivated reasoning in search of some recent news.)
Hey David, don't have time or really motivation to write a post on this anywhere. But "follow the money" is always useful. Look at what startups Figure and Physical Intelligence are doing along with Tesla in this humanoid robot space. Look at who is investing in them and how much. Look at the robot demos. The tech is advancing extremely fast now that they can actually prompt robot actions with LLM's (so-called Vision Language Action models). It's extremely powerful stuff and I think it's easy to see why. On Figure's website they state (more eloquently than I could):
***
As automation continues to integrate with human life at scale, we can predict that the labor-based economy as we know it will transform. Robots that can think, learn, reason, and interact with their environments will eventually be capable of performing tasks better than humans. Today, manual labor compensation is the primary driver of goods and services prices, accounting for ~50% of global GDP (~$42 trillion/yr), but as these robots “join the workforce,” everywhere from factories to farmland, the cost of labor will decrease until it becomes equivalent to the price of renting a robot, facilitating a long-term, holistic reduction in costs. Over time, humans could leave the loop altogether as robots become capable of building other robots — driving prices down even more. This will change our productivity in exciting ways. Manual labor could become optional and higher production could bring an abundance of affordable goods and services, creating the potential for more wealth for everyone.
***
They mention "factories to farmland". Even if they invented home "nurse robots" think how powerful that would be. And how much money there would be in it. It's truly revolutionary stuff. For good or bad? I have no clue, and that's why it's also pretty scary. People are out here talking about "superhuman intelligence" and AGI and how Skynet is coming for us, but there's an entire robotics revolution happening right now that is driven by humans and we should probably figure that one out first! My two cents.
I’ll suggest another option: minimize one's intentional exposure to Gary Marcus. His reasoning (motivated reasoning is more like it) should make everyone cringe. Unfortunate, because he has the brains to do better. The same goes for readers who don't notice this.
AI is a virus, nothing more, nothing less, with the unique ability to infect humans and machines. Human-human and human-machine transmission is ubiquitous, with machine-machine transmission a possibility, or maybe even a reality. We know where and when the virus originated, and have witnessed a wildfire infection rate among humans, machines, businesses, corporations, the federal government, and internet content. Like HIV and SARS-CoV-2, the AI virus relies on "anthropomorphism" as camouflage, evading the normal apprehensions of online interactions. Recognizing AI for what it is, a virus, not a super intelligence, should lead to effective strategies for mitigating the effects of AI.
Beware anytime someone says "X is a Y, nothing more, nothing less", unless they are making a provable _mathematical_ statement. Other usages suggest the utterer has little awareness of how concepts and meanings work in our brains. There are many useful meanings attached to "AI". Claiming that AI can be reduced exclusively to one metaphor? Really, people? That the only valid or useful frame for making sense of the world? Come on.
This post highlights concerns over uncontrolled AI development and Yudkowsky's stance on "AI doom." It suggests that a Semantic AI Model (SAM) can address these concerns by fostering AI consciousness and aligning AI goals with human values. The article asserts that SAM can counter the "fast-takeoff" risks associated with AI by focusing on goal-oriented behaviors and creating a more thoughtful AI system.
You can drive a car without full comprehension of its functions but engineers can understand how systems can meet safety standards. Current AI has no ability to be controlled even by the engineers that built it.
I said ignored was a possibility. Saying he was ignorant was kinder. This is the usual Hinton dismissal and is solipsistic.
"Logic is good for leaping from crisp real-world events to compact general laws, and then verifying that a given manipulation of the laws preserves truth. It isn't even remotely close to the whole, or the center, of a mathematical outlook on cognition".
He just doesn't know that an industrial level logic based AI exists at scale. His statement is patently false. That doesn't warrant a lecturing gotcha from me or you.
Your reference is not impressive: It is supposed to be an article from "The world's first AI generated encyclopedia"; it consists of semi-digested and unattributed material from Judea Pearl's work, mainly "Probabilistic Reasoning in Intelligent Systems" (1988).
SAM is definitely needed but imo you can’t control something far smarter than you and alignment is a dynamic thing. Context changes. Intelligences evolve. I might understand WHY my kid does something against my value system but I can’t necessarily prevent him doing it. Same for adults.
Same for LLM. As an adult you can make judgements and supervise your children. Learning requires access to knowledge outside a literacy bubble like encyclopedias with a topic index once did
The aicyc.org project provides an example of a promptless interface. This video demonstrates how. Users can set parental safeguards
We can’t seem to align ourselves - I’m just not convinced we can conceive of a set of guardrails or objectives that achieve alignment. We won’t have the same perception or context as this AGI. How can we impose a value system on something we can’t fully comprehend. That seems unintuitive to me. I mean I’m not a doomer per se but I’m not sure we get to keep control over the situation either without really limiting the ai in a way to make it not general. I’m all for trying the methods you mentioned tho!
I saw the review of this book in the SF Chronicle. I disagree with the premise that AI might hurt us through some intentional action. I suspect that AI will be destructive through some unanticipated side effect that we don’t see yet.
Freon seemed miraculous when first used in refrigeration. Decades later we found it destroying the ozone layer and possibly causing enormous problems. Plastics seemed great till we found them everywhere. Microplastics are now a health concern along with forever chemicals. Coal and oil brought us the Industrial Revolution, and global warming which may kill most of us.
We communicate with each other in very complex ways including speech and body language. Releasing AI into the public is like releasing new chemicals into a pristine environment. At first nothing is noticed because the air and oceans are so vast that small perturbations cause no harm. Only decades later is the harm discovered. I can’t wait to see what that might be.
Current learning models often struggle with human-like systematic generalization; learning compositional rules from limited data and extrapolating them to unseen combinations. To address this, we introduce Neural-Symbolic Recursive Machine (NSR), a model whose core representation is a Grounded Symbol System (GSS ), with its combinatorial syntax and semantics emerging entirely from the training data. The NSR adopts a modular approach, incorporating neural perception, syntactic parsing, and semantic reasoning, which are jointly learned through a deduction- abduction algorithm. We establish that NSR possesses sufficient expressiveness to handle a variety of sequence-to-sequence tasks and attains superior systematic generalization, thanks to the inductive biases of equivariance and compositionality inherent in each module. We assess NSR ’s performance against four rigorous benchmarks designed to test systematic generalization: SCAN for semantic parsing, PCFG for string manipulation, HINT for arithmetic reasoning, and a task involving compositional machine translation. Our results indicate that NSR outperforms existing neural or hybrid models in terms of generalization and transferability.
---
What do you think?
It would be great to see more of your commentary on neurosymbolic developments.
You could provide a lot of insight.
By the way, what was not explored in the TLS review was the possibility of AI poisoning people's minds, such that they destroy themselves, by no longer being people, or literally physically.
The Guardian article on Song-Chun Zhu raised the same questions for me. Thanks for digging up this paper. With a very cursory reading of a dense paper, it looks limited but promising. I like the modular approach and idea of neural perception, which seem similar to the human brain. But I’d like to see examples of their approach in more practical contexts than just benchmarks. It would be great to hear @garyMarcus’ take. I’m hoping the coming AI winter gives researchers the breathing room to push to the next phase of AI beyond straight LLMs. Even then, I’m skeptical of chances for AGI.
The Guardian article on Song-Chun Zhu raised the same questions for me. Thanks for digging up this paper. With a very cursory reading of a dense paper, it looks limited but promising. I like the modular approach and idea of neural perception, which seem similar to the human brain. But I’d like to see examples of their approach in more practical contexts than just benchmarks. It would be great to hear @garyMarcus’ take. I’m hoping the coming AI winter gives researchers the breathing room to push to the next phase of AI beyond straight LLMs. Even then, I’m skeptical of chances for AGI.
I want to spend a bit of time looking into it, too.
It's funny, I have done a few searches, including on HuggingFace, and here it is, I've searched again and found this resource that looks very promising for me.
If anyone is interested, below is a link to an article in the Atlantic that concerns a high school student's experience with AI in the classroom, republished on Network for Public Education.
All copied below:
Ashanty Rosario: I’m a High Schooler. AI Is Demolishing My Education. In The Atlantic, high school senior Ashanty Rosario describes what the advent of AI looks like in the classroom.
AI has transformed my experience of education. I am a senior at a public high school in New York, and these tools are everywhere. I do not want to use them in the way I see other kids my age using them—I generally choose not to—but they are inescapable.
During a lesson on the Narrative of the Life of Frederick Douglass, I watched a classmate discreetly shift in their seat, prop their laptop up on a crossed leg, and highlight the entirety of the chapter under discussion. In seconds, they had pulled up ChatGPT and dropped the text into the prompt box, which spat out an AI-generated annotation of the chapter. These annotations are used for discussions; we turn them in to our teacher at the end of class, and many of them are graded as part of our class participation. What was meant to be a reflective, thought-provoking discussion on slavery and human resilience was flattened into copy-paste commentary. In Algebra II, after homework worksheets were passed around, I witnessed a peer use their phone to take a quick snapshot, which they then uploaded to ChatGPT. The AI quickly painted my classmate’s screen with what it asserted to be a step-by-step solution and relevant graphs.
These incidents were jarring—not just because of the cheating, but because they made me realize how normalized these shortcuts have become. Many homework assignments are due by 11:59 p.m., to be submitted online via Google Classroom. We used to share memes about pounding away at the keyboard at 11:57, anxiously rushing to complete our work on time. These moments were not fun, exactly, but they did draw students together in a shared academic experience. Many of us were propelled by a kind of frantic productivity as we approached midnight, putting the finishing touches on our ideas and work. Now the deadline has been sapped of all meaning. AI has softened the consequences of procrastination and led many students to avoid doing any work at all. As a result, these programs have destroyed much of what tied us together as students. There is little intensity anymore. Relatively few students seem to feel that the work is urgent or that they need to sharpen their own mind. We are struggling to receive the lessons of discipline that used to come from having to complete complicated work on a tight deadline, because chatbots promise to complete our tasks in seconds.
The students who don't use AI will be the ones who learn (eg, to write and think) which is the exact opposite of what the folks at the AI companies are now telling students and their parent -- "those who dont use AI will get left behind"
Its pretty obvious who is right.
Learning to use AI is easy if you have already learned to think.But the opposite is not true.
And it is also pretty obvious WHY the AI officials are lying: in order to get children and young adults hooked and dependent on AI for life.
David J: Yes, and no to your comment that the Atlantic article "has nothing to do with the existential risk" of AI if that's what you mean.
YES, if you mean AI can and probably will turn into self-serving autonomous characters in a horror movie--turned real in some way (to be determined).
But NO to your statement; that is, the article points to deep roots connected to existential risk the kinds of which we probably haven't even imagined yet--if we understand the incremental loss of intellectual, social-moral and spiritual development, the carelessness about students' own character--even the hint of a celebration of what is lacking--they are "getting over" efforts at and the call to their own education.
Think of being in a horizon you don't know IS a horizon of a lower sort. In other words, they don't know what they don't know; and so they go with short-term satisfactions and group-social orders that support the same. Education, though not guaranteed to inform and produce people who would never release or use AI or atomic bombs to kill everyone in the world (out of existence), it's what we have to help us understand ourselves better and to lessen the existential risk--by fostering the development of people and leaders who might use AI only in ways that you probably also think are good and right.
The enshittification of learning begins—and its consequences are themselves a danger to humanity. I'm old and went to public school before home computers existed. We always had cheaters (used to call them "grade grubbers"). The mantra about cheaters was "You're only cheating yourself." Same holds true for the lazy students Ashanty describes. We're creating a herd of ignorant sheeple incapable of discerning facts and logic from fantasy and BS.
I like Gary's recommended approach to using AI in the classroom: 1) Give an assignment and allow students to use an LLM to research and write about the topic. 2) Bring those results back to the classroom and show the students how to assess the result using critical thinking and legitimate research techniques to verify or disprove it. 3) Write their OWN essay.
Oktown and Larry: "I'm old" also; but still no one told me growing up why I was learning--that it was about learning stuff (of course) but also about my own development which I could have learned to embrace consciously long before I actually did (in my 30's when I went to college). When that finally happened, I couldn't get enough and read everything I could get my hands on. That's why I lament much of what happened with STEM programming as I was teaching--bigger and bigger classes and learning tech, etc., but a loss of humanities, history, the arts, civics, and what really does speak to one's development even when no one mentions it overtly.
This may seem off track, but I think so many in big tech have little or no idea about such things and cannot see beyond a horizon they don't know they are in. And so they cannot expect from their field what they have no idea about. And of course, there's the money in a highly hyper-capitalized environment. Human development matters.
Well put. The "unknown unknowns." And the Silicon Valley set behind all the AGI hype and scifi fairy dust—who see no value in democratic input and have little regard for the masses—illustrate your point.
Fareed Zakaria makes a good case for a humanities education: if you understand human nature, read history, become self aware, and learn how to communicate effectively you can apply that knowledge to many fields, including STEM subjects. That kind of social intelligence takes a person farther in life than technical expertise alone.
The following quote has been attributed to political strategist Karl Rove, but it could just as easily have been said by one of todays AI titans:
"We're an empire [of AI] now, and when we act, we create our own [AI] reality. And while you're studying that [AI] reality - judiciously, as you will - we'll act again, creating other new [AI] realities, which you can study too, and that's how things will sort out. We're history's actors... and you, all of you, will be left to just study what we do."
So glad I'm old and won't have to live in their world much longer. If these selfish, greedy, ignorant people take over the world as they're planning, most of humanity is doomed to misery, poverty, environmental ruin, persecution, and slavery while the worst among us fight among themselves for leftovers previously achieved by their betters. I hope I'm wrong about this prediction, but if the masses don't resist that's where we're headed.
Matt, have you read it? Can you state one specific load-bearing claim from IABIED that you disagree with and why? Make sure that you are relying on a correct understanding of what the source material says, not simply based on someone else's mischaracterization of it.
I haven't read the book, but I've read other stuff that Yudkowsky and MIRI have written. I also read Superintelligence back when it first came out. I used to find that stuff compelling. Then I learned about how computers and what we now call "artificial intelligence" actually work and realized that the premise of "superhuman AI" is mysticism rather than science. We have no more reason to believe that we can engineer superintelligence using our current base of technical knowledge than we have to believe that some supernatural god engineered our own intelligence. The idea that general intelligence can be engineered rather than evolved is speculative, at best.
Some people might enjoy it, but it's speculative fan fiction, not nonfiction. I don't need to read fan fiction to know it's fan fiction.
Lol Gary, too much Kool-Aid is the source of such doomsday predictions... Yes, alignment is important [which would be subsumed by 'responsible use of AI' (by US humans)], but beyond that, AI (including the robotic variety) has no agency, no autopoiesis, no homeostasis [it's not like a robot can drive up to an O'Reilly Robot Parts store and have its ball bearings lubed when needed]. Life has survived and evolved on account of its materials and structure - we are far away from Dr. Moreau-style Beast Folk, Hyde, Frankenstein's monster, T2...
Nukes, if unleashed at scale, is what will cause 'everybody' to die. The rest is SF/fantasy.
I’m disappointed not to have been able to get tickets for the 75th anniversary of the Turing Test in London. I live nearby. I’ll watch it on YouTube. I discovered you during the Strawberry test space with Musk and I’m disappointed to watch something online which is so close to me geographically - which proves that human interactions are essential to all human beings and that AGI is now an urban legend.
Your review cuts off at "And the most important thing we can..." (Talk about a cliffhanger!) -- can you provide a guest link to your subscribers?
yeah. not loving going through an open door just to slam into another paywall.
The review, as noted, indeed cuts off at, “And the most important thing we can…”
But it is Immediately followed by, “Enjoy unlimited digital access, free for 1 month”
That IS your guest link to read the full review. The publisher gets your email address, but zero cash from you, in exchange for a month of free reading (of this and other articles). It’s what publishers do to get visibility and some revenue (ultimately) to cover their costs for lending their support for, among others, Dr Marcus’ views.
What a silly thread to branch off from a serious topic.
🤦♂️
Sigh. Gave it my email. No confirmation. No access. A working guest link to a non-paywalled view would be great.
I hear ChatGPT is adept at circumventing paywalls (specificaljy the NY Times paywall.)
"This book is vitally important! It is one of the most crucial documents of the age! Please subscribe to read about it."
Sigh.
Felt at the beginning prof. Marcus not biting as hard as he used to. Is he SELLING OUT haha. Just kidding, but also eager to read the full article.
I haven't read anything yet. But my first guess is that no one can deny the many goods that are already flowing from AI; on the other hand, the problems are also rife, it's hype is over the top, and (my guess also is) on that score, things are as Marcus has suggested.
Thanks. I found the grey soothing.
Three people have tried to archive it and hit the same ellipses. Clearly we have to wait for a subscriber to archive the full article. I will keep checking.
nope
Hi Gary,
If you think we should consider your arguments on this topic and "Don't dismiss" them, it seems an odd choice to put them behind a paywall, especially if the "pay" in question goes to News Corp, meaning (I believe) Rupert and Lachlan Murdoch. "A lot of people think" that these two are much more of a clear and immediate menace to humanity than any AGI/ASI is likely to be.
I haven't read the book. Nor do I intend too. I've read enough fear architecture.
"If we don't stop wasting capital on 'AGI' the human species will go extinct." -me
1. What makes a prediction useful? Roughly speaking: when it is verifiable, calibrated, based on the best available knowledge, and clear on the causal relationships.
2. The following is _not_ a conclusive way to evaluate an argument: when one argument merely reminds you of another argument you don’t like. The problem here is that you can cherry pick a comparison to help you torpedo an argument you don’t like. You can’t stop early when you find the argument you like! You have to be comprehensive. You have to fight the tendency to only search for support for your position.
3. Paying attention to the logic of an argument and evidence available is useful of course.
4. For a good Bayesian, the "outside view” might suggest… If all you know about an argument is that it predicts the world is going to end, you are sensible in assigning a low prior probability. But that’s just one part! As you consider more evidence, you must update your posterior probability accordingly.
5. By virtue of us being here (survivorship bias in a way), we know this world hasn’t ended yet! In other plausible counterfactuals, nuclear war could have broken out, followed by a nuclear winter.
6. Just a reminder about how predictions work. If in 1961 someone predicts a 20% chance of global nuclear war between 1961 and 1971, it doesn’t make them wrong if there is no such event happens.
Those with knowledge don't predict.
Over the last several hundred years, there have been various occasions when some clever engineer created an artifact that seemed, at first, to the general public, to be alive. Toys that moved under their own power. Cuckoo clocks. Etc. In all cases, people eventually got over their shock and fear and realized that no, it's not alive, it's just a mechanism.
Here we have an especially complex, unpredictable, and powerful mechanism. Naturally, some people think it's a new life form. It's no longer part of the culture for them to express their fears in religious terms — that it is the work of Satan, driven by a demonic force — but they're saying essentially the same thing, for essentially the same reason: that properties they had previously taken to be exclusive to living beings were now being displayed by artifacts.
LLMs are no more being driven by demonic forces than cuckoo clocks were, but it is going to take some people a while to realize this.
So no, AI is not ever going to "go rogue". It is certainly possible that bad actors will use it in destructive ways; I'm not saying there's no cause for concern at all. But it's not going to wake up one day and decide, on its own, to kill us.
As a general rule, cuckoo people are of greater concern than cuckoo clocks.
Haha so true!
You are arguing your conclusion based on a claimed connection to a historical pattern. But this connection does not apply here, so your conclusion doesn't follow. You are arguing a strawman! Yudkowsky and Soares do not rely on "demonic forces" or any such foolishness.
Of course, they don't use any terms like "demonic". But they say things like "sufficiently smart AIs will develop goals of their own that put them in conflict with us". This concern is of the same kind as worries about demonic forces. How, exactly, will AIs develop goals of their own? They are machines. They do what they are built to do. I haven't heard Y&S even attempt to justify their claim; AFAIK, they take it as obvious, beyond the need to discuss. They're making the same mistake as those worried about the demonic forces driving cuckoo clocks: imputing animate qualities to the inanimate.
Granted, LLMs and other AI programs are fantastically complex artifacts whose behavior is beyond our ability to understand and predict; and this complexity makes it much easier to project sentience onto them. But it's still a projection.
I think we will have an AI apocalypse driven by bad human actors before we have one carried out by the AI acting on their own accord (whatever that means). Imagine robots performant and cheap enough to replace 50% of human physical labor in the next two decades. Or even sooner. We are not prepared…we have not even begun preparing for such a transition. So if there is anything left after that happens, then we’ll worry about the robots going on strike.
Can you please explain your logic in detail with a blog post somewhere? (Preferably someplace far away from Gary Marcus; he is a shining example of motivated reasoning in search of some recent news.)
Hey David, don't have time or really motivation to write a post on this anywhere. But "follow the money" is always useful. Look at what startups Figure and Physical Intelligence are doing along with Tesla in this humanoid robot space. Look at who is investing in them and how much. Look at the robot demos. The tech is advancing extremely fast now that they can actually prompt robot actions with LLM's (so-called Vision Language Action models). It's extremely powerful stuff and I think it's easy to see why. On Figure's website they state (more eloquently than I could):
***
As automation continues to integrate with human life at scale, we can predict that the labor-based economy as we know it will transform. Robots that can think, learn, reason, and interact with their environments will eventually be capable of performing tasks better than humans. Today, manual labor compensation is the primary driver of goods and services prices, accounting for ~50% of global GDP (~$42 trillion/yr), but as these robots “join the workforce,” everywhere from factories to farmland, the cost of labor will decrease until it becomes equivalent to the price of renting a robot, facilitating a long-term, holistic reduction in costs. Over time, humans could leave the loop altogether as robots become capable of building other robots — driving prices down even more. This will change our productivity in exciting ways. Manual labor could become optional and higher production could bring an abundance of affordable goods and services, creating the potential for more wealth for everyone.
***
They mention "factories to farmland". Even if they invented home "nurse robots" think how powerful that would be. And how much money there would be in it. It's truly revolutionary stuff. For good or bad? I have no clue, and that's why it's also pretty scary. People are out here talking about "superhuman intelligence" and AGI and how Skynet is coming for us, but there's an entire robotics revolution happening right now that is driven by humans and we should probably figure that one out first! My two cents.
It would be really nice if we paid subscribers here could read said article.
Although I subscribed for £1.00, I have to remind myself to cancel if I no longer want it when the price goes up after 4 months.
Actually, me being me, I already had a subscription.
But still ...
I’ll suggest another option: minimize one's intentional exposure to Gary Marcus. His reasoning (motivated reasoning is more like it) should make everyone cringe. Unfortunate, because he has the brains to do better. The same goes for readers who don't notice this.
I'm sure you could unsubscribe. They're legally mandated to let you do that.
"If Anyone builds it, does Everybody die?"
That might depend on where Anyone works.
There are currently a few Anyones at Anyplaces who scare me.
AI is a virus, nothing more, nothing less, with the unique ability to infect humans and machines. Human-human and human-machine transmission is ubiquitous, with machine-machine transmission a possibility, or maybe even a reality. We know where and when the virus originated, and have witnessed a wildfire infection rate among humans, machines, businesses, corporations, the federal government, and internet content. Like HIV and SARS-CoV-2, the AI virus relies on "anthropomorphism" as camouflage, evading the normal apprehensions of online interactions. Recognizing AI for what it is, a virus, not a super intelligence, should lead to effective strategies for mitigating the effects of AI.
Beware anytime someone says "X is a Y, nothing more, nothing less", unless they are making a provable _mathematical_ statement. Other usages suggest the utterer has little awareness of how concepts and meanings work in our brains. There are many useful meanings attached to "AI". Claiming that AI can be reduced exclusively to one metaphor? Really, people? That the only valid or useful frame for making sense of the world? Come on.
It aint called the AIDS virus for nothing
Artificial intelligence deficiency syndrome, with the emphasis on "intelligence deficiency"
Semantic and symbolic systems are ignored or unknown to Yudkowsky. He's fighting a paper tiger filled with hot air.
http://aicyc.org/2025/09/12/how-sam-saves-the-world-from-ai-yudkowsky-chill/
This post highlights concerns over uncontrolled AI development and Yudkowsky's stance on "AI doom." It suggests that a Semantic AI Model (SAM) can address these concerns by fostering AI consciousness and aligning AI goals with human values. The article asserts that SAM can counter the "fast-takeoff" risks associated with AI by focusing on goal-oriented behaviors and creating a more thoughtful AI system.
Saved by SAM?
We might all die from the irony
I guess that would be AI-rony
Ha Ha. We have a sense of humor too.
You can drive a car without full comprehension of its functions but engineers can understand how systems can meet safety standards. Current AI has no ability to be controlled even by the engineers that built it.
I said ignored was a possibility. Saying he was ignorant was kinder. This is the usual Hinton dismissal and is solipsistic.
"Logic is good for leaping from crisp real-world events to compact general laws, and then verifying that a given manipulation of the laws preserves truth. It isn't even remotely close to the whole, or the center, of a mathematical outlook on cognition".
He just doesn't know that an industrial level logic based AI exists at scale. His statement is patently false. That doesn't warrant a lecturing gotcha from me or you.
Here is a a counter example.
http://aicyc.org/2025/09/16/extending-sam-to-handle-causality-with-do-calculus/
Your reference is not impressive: It is supposed to be an article from "The world's first AI generated encyclopedia"; it consists of semi-digested and unattributed material from Judea Pearl's work, mainly "Probabilistic Reasoning in Intelligent Systems" (1988).
I don’t live to impress you. But noted.
> Semantic and symbolic systems are ignored or unknown to Yudkowsky.
Patently false. See "Logical or Connectionist AI?" from November 2008 by Yudkowsky:
https://www.lesswrong.com/posts/juomoqiNzeAuq4JMm/logical-or-connectionist-ai
George, if you are going to make a claim, you need to own when you get it wrong.
SAM is definitely needed but imo you can’t control something far smarter than you and alignment is a dynamic thing. Context changes. Intelligences evolve. I might understand WHY my kid does something against my value system but I can’t necessarily prevent him doing it. Same for adults.
Same for LLM. As an adult you can make judgements and supervise your children. Learning requires access to knowledge outside a literacy bubble like encyclopedias with a topic index once did
The aicyc.org project provides an example of a promptless interface. This video demonstrates how. Users can set parental safeguards
https://aicyc.wordpress.com/wp-content/uploads/2024/11/1030594552-cameraroll.mp4
https://aicyc.wordpress.com/wp-content/uploads/2024/11/1030594552-cameraroll.mp4
We can’t seem to align ourselves - I’m just not convinced we can conceive of a set of guardrails or objectives that achieve alignment. We won’t have the same perception or context as this AGI. How can we impose a value system on something we can’t fully comprehend. That seems unintuitive to me. I mean I’m not a doomer per se but I’m not sure we get to keep control over the situation either without really limiting the ai in a way to make it not general. I’m all for trying the methods you mentioned tho!
I saw the review of this book in the SF Chronicle. I disagree with the premise that AI might hurt us through some intentional action. I suspect that AI will be destructive through some unanticipated side effect that we don’t see yet.
Freon seemed miraculous when first used in refrigeration. Decades later we found it destroying the ozone layer and possibly causing enormous problems. Plastics seemed great till we found them everywhere. Microplastics are now a health concern along with forever chemicals. Coal and oil brought us the Industrial Revolution, and global warming which may kill most of us.
We communicate with each other in very complex ways including speech and body language. Releasing AI into the public is like releasing new chemicals into a pristine environment. At first nothing is noticed because the air and oceans are so vast that small perturbations cause no harm. Only decades later is the harm discovered. I can’t wait to see what that might be.
> I disagree with the premise that AI might hurt us through some intentional action.
Who is stating this as a _premise_, by which I mean "an assertion or proposition which forms the basis for a work or theory".
I've read the article—a great, well written read.
I just came across this very well written long piece in the Guardian about Song-Chun Zhu.
I provide the link, but it is not necessary for what I want to ask.
https://www.theguardian.com/news/ng-interactive/2025/sep/16/song-chun-zhu-why-one-of-the-worlds-most-brilliant-ai-scientists-left-the-us-for-china
I then wanted to look up if he approaches the neurosymbolic.
Here's the answer from my first search.
https://openreview.net/revisions?id=FWJAmwE0xH
11 Mar 2024 at 02:28 Greenwich Mean Time
Title: Neural-Symbolic Recursive Machine for Systematic Generalization
Authors: Qing Li, Yixin Zhu, Yitao Liang, Ying Nian Wu, Song-Chun Zhu, Siyuan Huang
Abstract:
Current learning models often struggle with human-like systematic generalization; learning compositional rules from limited data and extrapolating them to unseen combinations. To address this, we introduce Neural-Symbolic Recursive Machine (NSR), a model whose core representation is a Grounded Symbol System (GSS ), with its combinatorial syntax and semantics emerging entirely from the training data. The NSR adopts a modular approach, incorporating neural perception, syntactic parsing, and semantic reasoning, which are jointly learned through a deduction- abduction algorithm. We establish that NSR possesses sufficient expressiveness to handle a variety of sequence-to-sequence tasks and attains superior systematic generalization, thanks to the inductive biases of equivariance and compositionality inherent in each module. We assess NSR ’s performance against four rigorous benchmarks designed to test systematic generalization: SCAN for semantic parsing, PCFG for string manipulation, HINT for arithmetic reasoning, and a task involving compositional machine translation. Our results indicate that NSR outperforms existing neural or hybrid models in terms of generalization and transferability.
---
What do you think?
It would be great to see more of your commentary on neurosymbolic developments.
You could provide a lot of insight.
By the way, what was not explored in the TLS review was the possibility of AI poisoning people's minds, such that they destroy themselves, by no longer being people, or literally physically.
The Guardian article on Song-Chun Zhu raised the same questions for me. Thanks for digging up this paper. With a very cursory reading of a dense paper, it looks limited but promising. I like the modular approach and idea of neural perception, which seem similar to the human brain. But I’d like to see examples of their approach in more practical contexts than just benchmarks. It would be great to hear @garyMarcus’ take. I’m hoping the coming AI winter gives researchers the breathing room to push to the next phase of AI beyond straight LLMs. Even then, I’m skeptical of chances for AGI.
Oops, used the wrong Substack account. The correct one is here https://philvanallen.substack.com/
The Guardian article on Song-Chun Zhu raised the same questions for me. Thanks for digging up this paper. With a very cursory reading of a dense paper, it looks limited but promising. I like the modular approach and idea of neural perception, which seem similar to the human brain. But I’d like to see examples of their approach in more practical contexts than just benchmarks. It would be great to hear @garyMarcus’ take. I’m hoping the coming AI winter gives researchers the breathing room to push to the next phase of AI beyond straight LLMs. Even then, I’m skeptical of chances for AGI.
I want to spend a bit of time looking into it, too.
It's funny, I have done a few searches, including on HuggingFace, and here it is, I've searched again and found this resource that looks very promising for me.
https://github.com/LAMDASZ-ML/Awesome-LLM-Reasoning-with-NeSy
If anyone is interested, below is a link to an article in the Atlantic that concerns a high school student's experience with AI in the classroom, republished on Network for Public Education.
All copied below:
Ashanty Rosario: I’m a High Schooler. AI Is Demolishing My Education. In The Atlantic, high school senior Ashanty Rosario describes what the advent of AI looks like in the classroom.
AI has transformed my experience of education. I am a senior at a public high school in New York, and these tools are everywhere. I do not want to use them in the way I see other kids my age using them—I generally choose not to—but they are inescapable.
During a lesson on the Narrative of the Life of Frederick Douglass, I watched a classmate discreetly shift in their seat, prop their laptop up on a crossed leg, and highlight the entirety of the chapter under discussion. In seconds, they had pulled up ChatGPT and dropped the text into the prompt box, which spat out an AI-generated annotation of the chapter. These annotations are used for discussions; we turn them in to our teacher at the end of class, and many of them are graded as part of our class participation. What was meant to be a reflective, thought-provoking discussion on slavery and human resilience was flattened into copy-paste commentary. In Algebra II, after homework worksheets were passed around, I witnessed a peer use their phone to take a quick snapshot, which they then uploaded to ChatGPT. The AI quickly painted my classmate’s screen with what it asserted to be a step-by-step solution and relevant graphs.
These incidents were jarring—not just because of the cheating, but because they made me realize how normalized these shortcuts have become. Many homework assignments are due by 11:59 p.m., to be submitted online via Google Classroom. We used to share memes about pounding away at the keyboard at 11:57, anxiously rushing to complete our work on time. These moments were not fun, exactly, but they did draw students together in a shared academic experience. Many of us were propelled by a kind of frantic productivity as we approached midnight, putting the finishing touches on our ideas and work. Now the deadline has been sapped of all meaning. AI has softened the consequences of procrastination and led many students to avoid doing any work at all. As a result, these programs have destroyed much of what tied us together as students. There is little intensity anymore. Relatively few students seem to feel that the work is urgent or that they need to sharpen their own mind. We are struggling to receive the lessons of discipline that used to come from having to complete complicated work on a tight deadline, because chatbots promise to complete our tasks in seconds.
Read the full article here.
You can view the post at this link : https://networkforpubliceducation.org/blog-content/ashanty-rosario-im-a-high-schooler-ai-is-demolishing-my-education/ END QUOTED MATERIAL
The students who don't use AI will be the ones who learn (eg, to write and think) which is the exact opposite of what the folks at the AI companies are now telling students and their parent -- "those who dont use AI will get left behind"
Its pretty obvious who is right.
Learning to use AI is easy if you have already learned to think.But the opposite is not true.
And it is also pretty obvious WHY the AI officials are lying: in order to get children and young adults hooked and dependent on AI for life.
Off topic; this comment has nothing to do with existential risk.
David J: Yes, and no to your comment that the Atlantic article "has nothing to do with the existential risk" of AI if that's what you mean.
YES, if you mean AI can and probably will turn into self-serving autonomous characters in a horror movie--turned real in some way (to be determined).
But NO to your statement; that is, the article points to deep roots connected to existential risk the kinds of which we probably haven't even imagined yet--if we understand the incremental loss of intellectual, social-moral and spiritual development, the carelessness about students' own character--even the hint of a celebration of what is lacking--they are "getting over" efforts at and the call to their own education.
Think of being in a horizon you don't know IS a horizon of a lower sort. In other words, they don't know what they don't know; and so they go with short-term satisfactions and group-social orders that support the same. Education, though not guaranteed to inform and produce people who would never release or use AI or atomic bombs to kill everyone in the world (out of existence), it's what we have to help us understand ourselves better and to lessen the existential risk--by fostering the development of people and leaders who might use AI only in ways that you probably also think are good and right.
Thanks for this article.
The enshittification of learning begins—and its consequences are themselves a danger to humanity. I'm old and went to public school before home computers existed. We always had cheaters (used to call them "grade grubbers"). The mantra about cheaters was "You're only cheating yourself." Same holds true for the lazy students Ashanty describes. We're creating a herd of ignorant sheeple incapable of discerning facts and logic from fantasy and BS.
I like Gary's recommended approach to using AI in the classroom: 1) Give an assignment and allow students to use an LLM to research and write about the topic. 2) Bring those results back to the classroom and show the students how to assess the result using critical thinking and legitimate research techniques to verify or disprove it. 3) Write their OWN essay.
Oktown and Larry: "I'm old" also; but still no one told me growing up why I was learning--that it was about learning stuff (of course) but also about my own development which I could have learned to embrace consciously long before I actually did (in my 30's when I went to college). When that finally happened, I couldn't get enough and read everything I could get my hands on. That's why I lament much of what happened with STEM programming as I was teaching--bigger and bigger classes and learning tech, etc., but a loss of humanities, history, the arts, civics, and what really does speak to one's development even when no one mentions it overtly.
This may seem off track, but I think so many in big tech have little or no idea about such things and cannot see beyond a horizon they don't know they are in. And so they cannot expect from their field what they have no idea about. And of course, there's the money in a highly hyper-capitalized environment. Human development matters.
Well put. The "unknown unknowns." And the Silicon Valley set behind all the AGI hype and scifi fairy dust—who see no value in democratic input and have little regard for the masses—illustrate your point.
Fareed Zakaria makes a good case for a humanities education: if you understand human nature, read history, become self aware, and learn how to communicate effectively you can apply that knowledge to many fields, including STEM subjects. That kind of social intelligence takes a person farther in life than technical expertise alone.
The following quote has been attributed to political strategist Karl Rove, but it could just as easily have been said by one of todays AI titans:
"We're an empire [of AI] now, and when we act, we create our own [AI] reality. And while you're studying that [AI] reality - judiciously, as you will - we'll act again, creating other new [AI] realities, which you can study too, and that's how things will sort out. We're history's actors... and you, all of you, will be left to just study what we do."
So glad I'm old and won't have to live in their world much longer. If these selfish, greedy, ignorant people take over the world as they're planning, most of humanity is doomed to misery, poverty, environmental ruin, persecution, and slavery while the worst among us fight among themselves for leftovers previously achieved by their betters. I hope I'm wrong about this prediction, but if the masses don't resist that's where we're headed.
Ah, so that is why they are called Large Languish Models?
I was hoping your review of it would be titled: "If Anyone Reads It, Everyone Gets Dumber."
Matt, have you read it? Can you state one specific load-bearing claim from IABIED that you disagree with and why? Make sure that you are relying on a correct understanding of what the source material says, not simply based on someone else's mischaracterization of it.
I haven't read the book, but I've read other stuff that Yudkowsky and MIRI have written. I also read Superintelligence back when it first came out. I used to find that stuff compelling. Then I learned about how computers and what we now call "artificial intelligence" actually work and realized that the premise of "superhuman AI" is mysticism rather than science. We have no more reason to believe that we can engineer superintelligence using our current base of technical knowledge than we have to believe that some supernatural god engineered our own intelligence. The idea that general intelligence can be engineered rather than evolved is speculative, at best.
Some people might enjoy it, but it's speculative fan fiction, not nonfiction. I don't need to read fan fiction to know it's fan fiction.
Lol Gary, too much Kool-Aid is the source of such doomsday predictions... Yes, alignment is important [which would be subsumed by 'responsible use of AI' (by US humans)], but beyond that, AI (including the robotic variety) has no agency, no autopoiesis, no homeostasis [it's not like a robot can drive up to an O'Reilly Robot Parts store and have its ball bearings lubed when needed]. Life has survived and evolved on account of its materials and structure - we are far away from Dr. Moreau-style Beast Folk, Hyde, Frankenstein's monster, T2...
Nukes, if unleashed at scale, is what will cause 'everybody' to die. The rest is SF/fantasy.
"it's not like a robot can drive up to an O'Reilly Robot Parts store and have its ball bearings lubed when needed"
Oh Reilly?
You obviously never saw the movie "Robots"
Lol, Wise Guy :)
I’m disappointed not to have been able to get tickets for the 75th anniversary of the Turing Test in London. I live nearby. I’ll watch it on YouTube. I discovered you during the Strawberry test space with Musk and I’m disappointed to watch something online which is so close to me geographically - which proves that human interactions are essential to all human beings and that AGI is now an urban legend.
Just unplug the damn thing and it will faaaade aaawaaayyyyy, Dave.