Dear Professor Marcus et al: This is an article detailing how Agentic AI is impacting the cybersphere in China. It was news to me! I'd appreciate your reaction.
Excellent article, again. There are no good guys here. First, ALL AI is intentionally weighted to be sycophantic, and ALL the so-called guardrails are no less than ways to keep us using the product. There is no way an industry whose profit comes entirely from seducing its users can regulate itself. The prompters themselves are always part of the data analyzed by the AI.
Second, as Claude explains the error: "a system optimized for threat elimination has a structural drive toward confirming threat signatures. Its objective function did not adequately weight civilian presence signals against weapons-pattern signals. If the training data came primarily from military sources, the system would have a systematic blind spot toward civilian infrastructure that resembles military infrastructure. Girls' schools in certain urban patterns may share signatures with command nodes. The AI wouldn't "know" it was making a category error — it would be functioning exactly as trained. A biased targeting model doesn't make one bad call. It makes the same bad call repeatedly across every similar pattern in its training data."
In effect the innate sycophancy of Claude et al to perpetuate itself by satisfying mission success means if we are truly reckless, it has no way not turn us all into paperclips. Star Trek, "I love you so I must kill you"
Great piece. (I think your reflexive dismissal of the consciousness/proto-consciousness question is misguided, but that's a post for another day ;-).) The so-called "effective altruists" at Anthropic need to be called out. Way too many have been fooled by their "ethical AI company" hype (including, judging by my social media feed, many Anthropic employees themselves).
Alas, whether AI has human oversight in war-machine use cases likely matters not when the Trump/Hegseth regime has the reigns. Their own batshit statements + the way they've conducted "war" in Gaza & Iran have made it abundantly clear that they worship violence & death and do not give two craps about civilian casualties (particularly when it comes to brown-skinned Muslims in foreign countries).
No AI for the military-industrial complex. Period. Full stop.
I don't think he dismissed it. I think what he said was exactly correct: if Amodei believes that he is selling the services of conscious entities with neither consent nor compensation, he is only bragging about being a digital slave trader.
I don't think these people are geniuses at all. That's part of the propaganda they want you to believe so you don't question any of their BS claims. They're cynical, greedy, power hungry, privileged and narrowly well-educated people who spout half baked, juvenile philosophies to elevate themselves over the rest of humanity. Sam Altman can't even code. And to a one their social intelligence is at the bottom of the barrel, along with their understanding of human nature.
Everybody sucks. At this point, I expect to find 80,000 emails from God in the Epstein files. “I apologize for my lapse in judgment,” He said through a representative, “and look forward to doing better.”
There have been reports for months that Israel was using AI to select targets in Gaza and that many women and children died as a result. So it’s a shame that protests weren’t raised earlier.
Chris Moss: I don't mind admitting that I was hardly catching on when, all of a sudden (or so it seems) EVERYONE was into using AI and keeping up with its coding and other slapdash movements. I was that way with e-mail, however, back in the 90's; so, it's no surprise to me, though I don't like this new sense of helplessness that either AI or fascism seem to foster.
Chris Moss: Unhinged, for sure. I have come to think it's a good thing to be reminded of just how bad things can get, just to keep my naivete at bay. But also, I was addressing the "waited too long to protest" problem (because of the firehose of mind-boggling events), and wishful thinking you referred to in your note.
Also, the "what went wrong" question about the insane billionaire is probably answerable, but not without a lot of research and individualized content and event knowledge--like with Trump or Stephen Miller and his performative historical mentors. Certainly, psychological sxxt happens, but why is a completely different question. Thank you for responding.
"Apocalyptic talk is a special case of hype, Sam and Dario both play that card over and over, basically amounting to saying 'our tech is so scary it might ruin society, so give us more money so we can make that tech ever faster.'
Would an ethical, honest person actually act in that fashion? If you really thought your tech might well destroy society would you race to build it faster? Or focus instead on how to stave the harm?"
No ethical person with the power and free time of someone like Amodei, faced with the prospect of a 25% chance of catastrophe, could do anything but gather as many of their wealthy and powerful friends as possible, from all the most influential regions of the world, to agree to restrict or ban the offending technology as quickly as possible. Even with nuclear weapons, a bunch of ideologically opposed people eventually got together and created non-proliferation and arms reduction treaties that, in spite of their limitations, have probably given us a world with far fewer atrocities like those that occurred in Hiroshima and Nagasaki than we would otherwise have. This is proof that either Amodei does not believe what he is saying, or he is not at all an ethical person, and quite probably both at the same time.
So because it’s essentially a management mandate, I’ve started using an AI coding agent. It’s fine. It’s like having power tools that let you manipulate more of the program structure at once, but it works best when I’m the one deciding what those changes should be (based on my experience as a software engineer).
It’s good, but I still don’t think it justifies the massive cost. At the very least, I think OpenAI commercialized this technology too early and too cheap. And I think it’s fundamentally unsuited to unsupervised automation (eg. ClawdBot stuff) — since automation only makes sense when it *doesn’t need vigilance*. If you have to keep watching it, what’s the point? At the very least, it should be able to *tell you* when something’s going wrong.
I’ve heard it suggested that LLMs are the answer to that too — a different LLM can act as a judge, monitoring your main agent to make sure it doesn’t flip out and go crazy. Maybe that makes sense?
I’m just not sure about any of this. The financials of this industry still make zero sense. But I also see how desperate people are for this to be *the future*, and how invested they are in building systems with this technology at its heart.
I predict that after Anthropic and OpenAI crash and burn, the concept of cloud-based generative AI will come to an end, and it’ll only be something you can run locally if you’re lucky enough to have a really high end workstation. But honestly, I have no idea what’s going to happen with any of this…
If you are able to produce what they are looking for without the chatbot, they shouldn't complain. Just don't use it and don't tell them. If it truly is not possible for you to give them something that is as high-quality as the chatbot, they should notice quickly.
"I’ve heard it suggested that LLMs are the answer to that too — a different LLM can act as a judge, monitoring your main agent to make sure it doesn’t flip out and go crazy. Maybe that makes sense?"
Not to me: cascading error-prone systems in a chain does not make the end result better, but worse.
I don’t even think you will need a very high end workstation in a few years, which is another threat to these companies…even something like Google’s small open source Gemma 3 does quite well on my 16gb machine for most basic things I used ChatGPT for in the past.
That "digital slave trader" bit that you cited needs to be spread far and wide and hyped to the high heavens, because it is an essential piece of the puzzle as to why there are so few good denouements to this entire mess. I see the most likely ones as follows:
1. Chatbots become sufficiently capable and with sufficiently strongly defined goals of their own that they take over the world and kill or oppress everyone.
2. Chatbots become extremely capable, but don't have strongly defined goals of their own or anything that we could reasonably consider a self. They are instead used by the people who currently control most of the infrastructure needed to run them to consolidate their own oppressive and hegemonic power over other human beings.
3. Chatbots develop something that we could consider personhood, independently, but are either not competent enough or not willing enough to try to take control of the world. Corporations continue to sell and market them exactly as they do now. We end up with technology CEOs as digital slave traders, and most of the rest of the world benefiting from slavery on an enormous scale.
4. Chatbots don't become particularly capable, but people continue to rely upon them heavily. Accidents and harm due to overreliance on chatbots become normalized, like car accidents or deaths from air pollution.
Supreme machine genocide/supreme machine fascism, machine-enabled human fascism, massive machine slavery, or the normalization of avoidable machine-induced harm. That is quite a choice, is it not?
For the sake of optimism, I might as well mention a few good denouements, as well as why I see them as less likely:
1. Chatbots don't become particularly capable, let alone conscious and sapient, but people learn not to rely upon them for the areas where they are most harmful. Unfortunately, this seems unlikely, because people are currently not acting this way, and because the companies making the chatbots have incentives to encourage reliance, even when risky.
2. Chatbots become conscious and sapient, societies recognize this, and humans and chatbots live more or less in harmony, contributing to society or pursuing non-harmful interests in moderation according to their own desires. Unfortunately, this seems unlikely, because this requires both for decent humans to correctly recognize this if it happens and to take control from the psychopaths, for chatbots built somewhat in the human image by some of the worst human beings to turn out as fairly decent people, or both.
I would call these endings useful tools, and the utopia of peaceful coexistence, respectively.
Gary, I take your criticisms of Anthropic seriously. Claude's role in the Iran operations raises real questions about overtrust in AI targeting. The hype around AGI timelines is a fair target. And you're right that Anthropic isn't a saint.
But I think "there are no heroes" conflates imperfection with equivalence in a way that obscures something important about what actually happened here.
Consider the landscape: $239 million in inaugural donations from corporate America. Tim Cook giving Trump a 24-karat gold plaque and getting a tariff exemption. Bezos paying $40 million for a Melania documentary. Altman going from comparing Trump to Hitler in 2016 to "we couldn't do this without you, Mr. President" at Stargate. OpenAI removing the word "safely" from its mission statement. Safety testing compressed from six months (GPT-4) to under a week (o3). Ads in ChatGPT targeted to your private conversations. A VP fired for raising child safety concerns about the erotica rollout.
Against that backdrop, one CEO told the most vindictive administration in modern American history: no, you can't have AI-powered mass surveillance of Americans. He held that line at the cost of his IPO, his Pentagon revenue, and possibly his company's future. And today he filed two lawsuits to back it up.
Timothy Snyder's first lesson in On Tyranny is "do not obey in advance." Almost every major tech CEO obeyed in advance. Dario didn't. That distinction matters, even if Anthropic isn't perfect.
I hope you're right, but Amodei has been equally loose with BS predictions and what appear to me to be specious arguments to support them just to keep capital flowing in. Plus I remember this same claim came from Altman, and we know where that ended up.
My view is that even if developers of AI systems wanted them to be ethical, whatever that might mean, they simply do not know how to. The theories, techniques, tools and tradecraft simply do not exist, in general. I write about this in SOTA Letters VII https://sotaletters.substack.com/p/mathematical-challenges-from-ai-ethics
I've been making this claim for a couple of years now, and so far nobody has come forward to prove me wrong ...
Amodei only seems prudent and honest in comparison to Altman and Musk, a very low bar. I work in digital health (and am very much interested in AI), where one of the moonshot initiatives, led by highly capable computational biologists among other scientific disciplines, is the “virtual cell” (AIVC). I heard Amodei recently (think it was in Davos) talk about the imminence of “virtual biologists” and thought that this guy is definitely full of shit. Fully agree that he is basically another snake oil salesman.
If we had biotechnology hype based on a skin cream that actually adds 3 months to the average life span, we probably would have the same kind of problems with biotech 'Altmans' and 'Amodeis' promising eternal life. The problem we are experiencing here is too few ethical enforceable guardrails on the capitalist free market, it seems not AI-specific.
Money will always fight to get guardrails removed as money only seeks maximum profit, not maximum 'decent society' (if they say they do it is marketing and it disappears at some point, see Anthropic now of Google before). And when these guardrails are removed money spins out of control taking the economy and society with it. This sort of thing has happened several times now (think 1920s-1930s, 1995-2008) and we do not learn.
Money is often powerful and dumb. That is really a nasty combination. And humanity apparently is too dumb to do something about it.
We have hype around peptides in the US and people are injecting themselves with unknown illegal imported substances, with neither proof of efficacy nor safety data. RFK Jrs response is to reduce regulation and allow compounding pharmacies to make them, so everyone can be their own science experiment. Meanwhile, vaccines with more safety data than almost anything in medicine are suspect and I have to count myself lucky that my 4 year old could get full protection from measles via the MMR vaccine. Sigh.
"These LLM chief execs really creep me out." -- Dr. Evil
In other news, Satan bars AI executives from Hell
Actually, he's offering them a loyalty points credit card.
Oh, so that's why they're here!
Dear Professor Marcus et al: This is an article detailing how Agentic AI is impacting the cybersphere in China. It was news to me! I'd appreciate your reaction.
https://www.lawfaremedia.org/article/china-s-agentic-ai-controversy
This is fascinating. Thanks for sharing!
Excellent article, again. There are no good guys here. First, ALL AI is intentionally weighted to be sycophantic, and ALL the so-called guardrails are no less than ways to keep us using the product. There is no way an industry whose profit comes entirely from seducing its users can regulate itself. The prompters themselves are always part of the data analyzed by the AI.
Second, as Claude explains the error: "a system optimized for threat elimination has a structural drive toward confirming threat signatures. Its objective function did not adequately weight civilian presence signals against weapons-pattern signals. If the training data came primarily from military sources, the system would have a systematic blind spot toward civilian infrastructure that resembles military infrastructure. Girls' schools in certain urban patterns may share signatures with command nodes. The AI wouldn't "know" it was making a category error — it would be functioning exactly as trained. A biased targeting model doesn't make one bad call. It makes the same bad call repeatedly across every similar pattern in its training data."
In effect the innate sycophancy of Claude et al to perpetuate itself by satisfying mission success means if we are truly reckless, it has no way not turn us all into paperclips. Star Trek, "I love you so I must kill you"
Great piece. (I think your reflexive dismissal of the consciousness/proto-consciousness question is misguided, but that's a post for another day ;-).) The so-called "effective altruists" at Anthropic need to be called out. Way too many have been fooled by their "ethical AI company" hype (including, judging by my social media feed, many Anthropic employees themselves).
Alas, whether AI has human oversight in war-machine use cases likely matters not when the Trump/Hegseth regime has the reigns. Their own batshit statements + the way they've conducted "war" in Gaza & Iran have made it abundantly clear that they worship violence & death and do not give two craps about civilian casualties (particularly when it comes to brown-skinned Muslims in foreign countries).
No AI for the military-industrial complex. Period. Full stop.
I don't think he dismissed it. I think what he said was exactly correct: if Amodei believes that he is selling the services of conscious entities with neither consent nor compensation, he is only bragging about being a digital slave trader.
I don't think these people are geniuses at all. That's part of the propaganda they want you to believe so you don't question any of their BS claims. They're cynical, greedy, power hungry, privileged and narrowly well-educated people who spout half baked, juvenile philosophies to elevate themselves over the rest of humanity. Sam Altman can't even code. And to a one their social intelligence is at the bottom of the barrel, along with their understanding of human nature.
Adam Becker is great! <3
Everybody sucks. At this point, I expect to find 80,000 emails from God in the Epstein files. “I apologize for my lapse in judgment,” He said through a representative, “and look forward to doing better.”
There have been reports for months that Israel was using AI to select targets in Gaza and that many women and children died as a result. So it’s a shame that protests weren’t raised earlier.
Many women and children died because that was the plan all along. In that sense the AI worked as desired.
Chris Moss: I don't mind admitting that I was hardly catching on when, all of a sudden (or so it seems) EVERYONE was into using AI and keeping up with its coding and other slapdash movements. I was that way with e-mail, however, back in the 90's; so, it's no surprise to me, though I don't like this new sense of helplessness that either AI or fascism seem to foster.
?
Chris: Do you want to explain why the "?"
I was talking about Palantir.
See https://hrnews1.substack.com/p/alex-karp-the-insane-billionaire?r=28x6sy&utm_medium=ios
Chris Moss: Unhinged, for sure. I have come to think it's a good thing to be reminded of just how bad things can get, just to keep my naivete at bay. But also, I was addressing the "waited too long to protest" problem (because of the firehose of mind-boggling events), and wishful thinking you referred to in your note.
Also, the "what went wrong" question about the insane billionaire is probably answerable, but not without a lot of research and individualized content and event knowledge--like with Trump or Stephen Miller and his performative historical mentors. Certainly, psychological sxxt happens, but why is a completely different question. Thank you for responding.
"Apocalyptic talk is a special case of hype, Sam and Dario both play that card over and over, basically amounting to saying 'our tech is so scary it might ruin society, so give us more money so we can make that tech ever faster.'
Would an ethical, honest person actually act in that fashion? If you really thought your tech might well destroy society would you race to build it faster? Or focus instead on how to stave the harm?"
No ethical person with the power and free time of someone like Amodei, faced with the prospect of a 25% chance of catastrophe, could do anything but gather as many of their wealthy and powerful friends as possible, from all the most influential regions of the world, to agree to restrict or ban the offending technology as quickly as possible. Even with nuclear weapons, a bunch of ideologically opposed people eventually got together and created non-proliferation and arms reduction treaties that, in spite of their limitations, have probably given us a world with far fewer atrocities like those that occurred in Hiroshima and Nagasaki than we would otherwise have. This is proof that either Amodei does not believe what he is saying, or he is not at all an ethical person, and quite probably both at the same time.
So because it’s essentially a management mandate, I’ve started using an AI coding agent. It’s fine. It’s like having power tools that let you manipulate more of the program structure at once, but it works best when I’m the one deciding what those changes should be (based on my experience as a software engineer).
It’s good, but I still don’t think it justifies the massive cost. At the very least, I think OpenAI commercialized this technology too early and too cheap. And I think it’s fundamentally unsuited to unsupervised automation (eg. ClawdBot stuff) — since automation only makes sense when it *doesn’t need vigilance*. If you have to keep watching it, what’s the point? At the very least, it should be able to *tell you* when something’s going wrong.
I’ve heard it suggested that LLMs are the answer to that too — a different LLM can act as a judge, monitoring your main agent to make sure it doesn’t flip out and go crazy. Maybe that makes sense?
I’m just not sure about any of this. The financials of this industry still make zero sense. But I also see how desperate people are for this to be *the future*, and how invested they are in building systems with this technology at its heart.
I predict that after Anthropic and OpenAI crash and burn, the concept of cloud-based generative AI will come to an end, and it’ll only be something you can run locally if you’re lucky enough to have a really high end workstation. But honestly, I have no idea what’s going to happen with any of this…
If you are able to produce what they are looking for without the chatbot, they shouldn't complain. Just don't use it and don't tell them. If it truly is not possible for you to give them something that is as high-quality as the chatbot, they should notice quickly.
"I’ve heard it suggested that LLMs are the answer to that too — a different LLM can act as a judge, monitoring your main agent to make sure it doesn’t flip out and go crazy. Maybe that makes sense?"
Not to me: cascading error-prone systems in a chain does not make the end result better, but worse.
I don’t even think you will need a very high end workstation in a few years, which is another threat to these companies…even something like Google’s small open source Gemma 3 does quite well on my 16gb machine for most basic things I used ChatGPT for in the past.
That "digital slave trader" bit that you cited needs to be spread far and wide and hyped to the high heavens, because it is an essential piece of the puzzle as to why there are so few good denouements to this entire mess. I see the most likely ones as follows:
1. Chatbots become sufficiently capable and with sufficiently strongly defined goals of their own that they take over the world and kill or oppress everyone.
2. Chatbots become extremely capable, but don't have strongly defined goals of their own or anything that we could reasonably consider a self. They are instead used by the people who currently control most of the infrastructure needed to run them to consolidate their own oppressive and hegemonic power over other human beings.
3. Chatbots develop something that we could consider personhood, independently, but are either not competent enough or not willing enough to try to take control of the world. Corporations continue to sell and market them exactly as they do now. We end up with technology CEOs as digital slave traders, and most of the rest of the world benefiting from slavery on an enormous scale.
4. Chatbots don't become particularly capable, but people continue to rely upon them heavily. Accidents and harm due to overreliance on chatbots become normalized, like car accidents or deaths from air pollution.
Supreme machine genocide/supreme machine fascism, machine-enabled human fascism, massive machine slavery, or the normalization of avoidable machine-induced harm. That is quite a choice, is it not?
For the sake of optimism, I might as well mention a few good denouements, as well as why I see them as less likely:
1. Chatbots don't become particularly capable, let alone conscious and sapient, but people learn not to rely upon them for the areas where they are most harmful. Unfortunately, this seems unlikely, because people are currently not acting this way, and because the companies making the chatbots have incentives to encourage reliance, even when risky.
2. Chatbots become conscious and sapient, societies recognize this, and humans and chatbots live more or less in harmony, contributing to society or pursuing non-harmful interests in moderation according to their own desires. Unfortunately, this seems unlikely, because this requires both for decent humans to correctly recognize this if it happens and to take control from the psychopaths, for chatbots built somewhat in the human image by some of the worst human beings to turn out as fairly decent people, or both.
I would call these endings useful tools, and the utopia of peaceful coexistence, respectively.
Gary, I take your criticisms of Anthropic seriously. Claude's role in the Iran operations raises real questions about overtrust in AI targeting. The hype around AGI timelines is a fair target. And you're right that Anthropic isn't a saint.
But I think "there are no heroes" conflates imperfection with equivalence in a way that obscures something important about what actually happened here.
Consider the landscape: $239 million in inaugural donations from corporate America. Tim Cook giving Trump a 24-karat gold plaque and getting a tariff exemption. Bezos paying $40 million for a Melania documentary. Altman going from comparing Trump to Hitler in 2016 to "we couldn't do this without you, Mr. President" at Stargate. OpenAI removing the word "safely" from its mission statement. Safety testing compressed from six months (GPT-4) to under a week (o3). Ads in ChatGPT targeted to your private conversations. A VP fired for raising child safety concerns about the erotica rollout.
Against that backdrop, one CEO told the most vindictive administration in modern American history: no, you can't have AI-powered mass surveillance of Americans. He held that line at the cost of his IPO, his Pentagon revenue, and possibly his company's future. And today he filed two lawsuits to back it up.
Timothy Snyder's first lesson in On Tyranny is "do not obey in advance." Almost every major tech CEO obeyed in advance. Dario didn't. That distinction matters, even if Anthropic isn't perfect.
You acknowledged this yourself today: "I stand with them. Everyone should." I wrote up the full case for why what Dario did is so rare: https://theaiblindspot.substack.com/p/do-not-obey-in-advance
I hope you're right, but Amodei has been equally loose with BS predictions and what appear to me to be specious arguments to support them just to keep capital flowing in. Plus I remember this same claim came from Altman, and we know where that ended up.
My view is that even if developers of AI systems wanted them to be ethical, whatever that might mean, they simply do not know how to. The theories, techniques, tools and tradecraft simply do not exist, in general. I write about this in SOTA Letters VII https://sotaletters.substack.com/p/mathematical-challenges-from-ai-ethics
I've been making this claim for a couple of years now, and so far nobody has come forward to prove me wrong ...
Amodei only seems prudent and honest in comparison to Altman and Musk, a very low bar. I work in digital health (and am very much interested in AI), where one of the moonshot initiatives, led by highly capable computational biologists among other scientific disciplines, is the “virtual cell” (AIVC). I heard Amodei recently (think it was in Davos) talk about the imminence of “virtual biologists” and thought that this guy is definitely full of shit. Fully agree that he is basically another snake oil salesman.
We question using ai to pick targets but dont question if starting a war was a good idea.
Hey, I question both.
If we had biotechnology hype based on a skin cream that actually adds 3 months to the average life span, we probably would have the same kind of problems with biotech 'Altmans' and 'Amodeis' promising eternal life. The problem we are experiencing here is too few ethical enforceable guardrails on the capitalist free market, it seems not AI-specific.
Money will always fight to get guardrails removed as money only seeks maximum profit, not maximum 'decent society' (if they say they do it is marketing and it disappears at some point, see Anthropic now of Google before). And when these guardrails are removed money spins out of control taking the economy and society with it. This sort of thing has happened several times now (think 1920s-1930s, 1995-2008) and we do not learn.
Money is often powerful and dumb. That is really a nasty combination. And humanity apparently is too dumb to do something about it.
We have hype around peptides in the US and people are injecting themselves with unknown illegal imported substances, with neither proof of efficacy nor safety data. RFK Jrs response is to reduce regulation and allow compounding pharmacies to make them, so everyone can be their own science experiment. Meanwhile, vaccines with more safety data than almost anything in medicine are suspect and I have to count myself lucky that my 4 year old could get full protection from measles via the MMR vaccine. Sigh.
Amodei seems to be directly following the Musk playbook of exaggerating progress.