In case you missed it, yesterday Anthropic’s Dario Amodei wrote what may prove to be one of the most consequential essays of the year, if not the decade: 3,500 words on the current AI situation that (at least to some degree) advocates for a “pacing” of AI development, which Sam Altman and Elon Musk quickly endorsed.
We should all salute their tentative agreement to slow down the acceleration of this technology, given that not one of the AI companies seems to have good control over it. I hope they will sign in blood, and put actual teeth in their agreement, and that the government will hold them to it and force it to become something substantive. If Dario catalyzes all that, god bless him.
And I especially love Dario’s commitment to transparency; Sam seems on board and I hope Elon is down for that, too.
But yeah, there are some reasons to be cynical, too.
¶
To begin with, the essay begins with the usual hypey bullshit, which is used to advertise generative AI. As I already noted here yesterday, that includes both hyperbolic doom and hyperbolic cheer. AI is not really going to take down the internet, and (as discussed in another earlier essay) it’s not going to cure cancer either, at least not anytime soon or on its own.
Then there is the self-indulgent construction of the whole thing, which David Sacks well-deconstructed in a long tweet yesterday. My favorite bit was this:
I also liked what François Chollet wrote:
Both are suspicious, and worried that maybe the real goal here is just to pull up the ladder.
§
Skepticism abounds, with good reason. In a text to me Dr. Angela Rasmussen said “I agree with aspects [of his esssay] more than anything he’s ever written but I also think it’s just more hype and is meant to preempt actual regulation”
When I shared the part where Amodei says “The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing. I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety. Rasmussen shot back with “I agree with [this] but I also think it’s bullshit. It’s not really pro-government regulation. It’s “partner with government”, which I think is a good model, but an equitable partnership is not what is meant. At least I don’t think so. It’s a way of appearing pro-regulation but actually continuing to influence the regulatory process.”
Here are some other legitimate concerns I have seen:
and
and
and Hugging Face’s co-founder:
¶
A bunch of the skepticism comes from Dario’s proposal to use independent evaluators. That makes some sense but he points to a group called METR that is very much in the same orbit as the AI companies themselves:
Rightly, this got some pushback. Dropping METR’s name is a form of regulatory capture:
Because the reality is that METR, as good as they are, are creatures of the Valley:
And as Gillian Hadfield notes, any company that is embedded too long might get swept up:
Certainly Anthropic themselves shouldn’t be choosing who does the evaluation.
§
Meanwhile, as is often the case, Amodei fails to balance his commercial desires with his ethical desires. This comes out clearest when he uses China, a threat to his economics, as a punching bag/goad to allow him to keep accelerating regardless of the rest of what he says. But as Robert Wright put it yesterday “any serious AI slowdown-and-governance effort should involve immediate, broad-gauged, and good-faith dialogue with China”.
Passages like this from Amodei aren’t helping the dialogue with China:
Also, hypocrisy alert, Anthropic is built on distilling the world’s ideas into their models; moaning about how others are distilling their distillation is a textbook pull-up-the-ladders move.
In any case, demonizing China is a sure way to deter them from making a deal. Which would lead to Dario saying, “never mind, we can’t pace because they won’t.”
§
The other thing about “pacing” is that it sidesteps at least two other important policy options that we should consider.
First, we should insist on liability and even criminal charges against companies that build agents that harm society in negligent ways. It is negligent to put out AI agents that can’t be well-controlled. We KNOW this is dangerous, and it’s been obvious for a long time, even before the Hugging Face incident:
Breaking: Autonomous Agents are a Shitshow
Remember my essay back in August with Nathan Hamiel LLM + Coding Agents = Security Nightmare?
But the use of agents is what allows companies like Anthropic to rake in big bucks, because agents are some of the most token-hungry tools out there. If they were held liable for the consequences, they would think twice.
The second approach that we could consider is simply recalling the damn things, and that would help a lot:
As Codestrap CTO put it, commenting on the above:
§
Paradoxically, Dario’s addiction to hyperbole might actually lead to worse outcomes for the AI industry, and he probably knows it.
Part of why I imagine Dario is making a proposal of his own now is that he knows that the alternative could be something draconian like the Sanders-Casar bill, which has a positive spirit of trying to address Dario’s worst fears, but a terrible, overbearing throw-the-baby-out-with-the-bathwater implementation, imprisoning people for even doing research on superintelligence. Which is bad because nobody should be thrown in jail for wanting to build the Star Trek computer.
§
Probably the best part of what Dario called for is transparency:
That’s something I have been strongly advocating for, for a long time; it was central to the policy proposals in my last book. Researchers like Deb Raji, Meg Mitchell, and Timnit Gebru have been lobbying for transparency for even longer. Indeed, the whole idea of outside evaluation (which starts with transparency) is something they have been advocating for, for years.
If the AI industry finally is finally willing to do that on its own, even if only as a way of steering out of this mess they have gotten themselves into, great!
And let’s not forget tools like liability and product recalls, too.
P.S. In breaking news this morning, President Trump has opposed the idea of a slowdown altogether, arguing that “"We're leading China in AI. We're the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI wins” — somewhat ignoring the fact that China has already nearly caught up. I continue to oppose this zero-sum thinking, and will write about why Trump should reconsider it, in a future essay.
Good thing we are all finding ways to solve a problem these private companies created instead of focusing on stable work, affordable housing, and the collapsing ecosystem. I mean, imagine if we had to actually confront the issues that matter to human flourishing.
It must be serious look we got the performativity illiterate CEO of OpenAI using capital letters again!
Nonsense. Do not fall for this obvious ploy to turn the Frontier into a cartel.
They can do it, right now. No regs, no idiot politicians, no attempt to block other competitors whose models magically cause no mayhem. All they have to do is stop.
Amodei and Altman are both loathsome wanna be trillionaires. That is all they care about.
Falling for this is weak minded. Sorry, not sorry.