Anderson Cooper looks pretty worried here, as Jacob Coxon, the ex-OpenAI/ex-Anthropic employee who is the media darling of the moment, tells us that the human species might not be here in five years.
I am here to tell you that Anderson Cooper doesn’t need to worry about that particular scenario, and that you don’t either.
§
Before I get there, I want to set some things straight. A bunch of people, apparently on the right, are trying to assassinate Coxon’s character. I have seen lies about how long he worked at Anthropic, possibly derived from an inaccurate AI-generated profile of him, and a weaponized thread characterizing him (without real evidence) as a Democratic operative. Someone described him as entry-level employee, which is absurd, given that he had already worked at OpenAI on a technical team for over two years.
In reality, we need a little nuance here; some of what Coxon says is true, some is speculative; some he is in a position to speak to, and some is out of his expertise. (The media too rarely distinguishes).
What I liked about Coxon’s now-infamous thread is that it cast light on what people at OpenAI and Anthropic are thinking. Since he worked there for a total of about three years (mainly at OpenAI, but most recently in Anthropic), he’s presumably in a position to testify to what might be common (presumably not universal) thinking at those companies.
The most important of his thread was the part that went towards state of mind:
rightly accusing the frontier companies of hubris:
Corroborating evidence soon came from Evan Hubinger, a current Anthropic employee:
And others eventually weighed in; OpenAI’s chief scientist actually said something similar. There’s good reason to believe that at least some companies really believe this kind of thing, and also (terrifyingly) that many have come up with justifications for believing that it is ok to continue working on a technology that they themselves appear to be dangerous.
But … there is less reason to believe that people at these companies have broad expertise about how the world works beyond their technical expertise.
Indeed, there is good reason to doubt that so-called Doomers have any clue what they are doing in terms of enacting change in the real world, or even in thinking through the consequences of their own actions. They are fabulous at getting publicity (the Yudkowsky/Soares bestseller, Bostrom’s bestseller, Coxon’s thread, the Pause letter, and so on). And they are fabulous at getting truly massive funding, perhaps close to (if not over) a billion dollars by now.
But in my view, the net effect of the last decade of “Doomer” talk (Coxon appears to be a card-carrying member of that party) has been to make the Frontier AI companies wealthier and more powerful. In so doing they have also aided and abetted a concentration of power that is in itself of highly dangerous. Everytime they scream that AI is going to kill us all, VCs toss in billions more. None of the AI safety work Doomers have sponsored has led to a demonstrated, implemented solution to the core problem. Enabling OpenAI and Anthropic may prove to be one of the worst things humanity has ever done, and they played a big role in that with their constant drama.
Furthermore, Doomers seem to constantly be selling humanity short, as I will discuss below. And (as also discussed below) they also seem to know not the slightest thing about how war works—which matters since they are implicitly or explicitly imagining a war-to-beat-all-wars between humans and machines.
Even if you believed that machines were trying to take over the world (which I don’t, at least not at present nor anytime soon), and that they were superintelligent (also not true yet, though presumably true eventually), and even if you suspended disbelief about unplugging the machines, you still have to think through how it is that the combination of motive and superintelligence would lead to the complete annihilation of the human species.
I have never, ever seen anyone from that world address that latter question with nuance and sophistication. Certainly Coxon has (so far) not done so.
§
The best attempt I have seen, such as it is, is a book that was wildly popular a year ago, Eliezer Yudkowsky and Nate Soares’ If Anyone Builds It, Everybody Dies.
The book has a lot going for it, both as a work of entertaining science fiction, and as a call to arms to get people to take AI risk seriously.
But the part in which it paints how AI might actually extinguish humanity is weak, menadering, and unconvincing. The New York Times Book Review dismissed it as being like scientology. I was kinder in the Time Literary Supplement, and much less dismissive, taking the details of the book far more seriously, but in the end the book’s argument was flaweed, for a multiple reasons.
Here are some excerpts of from what I wrote then, a year ago, all still quite relevant.
The first sums up the Yudkowsky/Soares argument. I presume that Coxon (who hasn’t made his argument explicit as far as I know) is relying on something similar:
The bad news is that Premise 1 (sorry I can’t handle the British spelling) is almost certainly true, though I seriously doubt it will happen by 2030. (Lots of well-known AI researchers like Rich Sutton and Yann LeCun and probably Ilya Sutskever and Fei-Fei Li would agree with me there; we all think that the field needs more breakthroughs before we reach superintelligence.)
That’s already reason for Anderson Cooper to breathe a sigh of relief; we have at least a bit more time before superintelligence arrives than Coxon seemed to allow. (Note that Transformers took 9 years to reach the current state; even if there a new breakthrough now it might take years to fully develop. And we probably need more than one, as I will discuss in a future essay.)
The good news for humanity is that the second and third premises of the Yudkowosky/Soares argument are far, far more shaky. Quoting what I said last year in TLS:
[Note that the Open AI Hugging Face incident was provoked as part of a training exercise, with guard rails partly turned off, and not something that happened organically and spontaneously.]
If the second premise seems unrealistic with respect to machines, the third seems unrealistic with respect to humans:
I sent the review to Yudkowsky, but never heard back. So far I know all my points about human resistance still stand.
Certainly Coxon has not (thus far) added anything substantive exto the argument.
In my view, the chance that AI will eliminate humans in the next five years is all but indistinguishable from zero. Humans are too geographically spread out, too genetically diverse, and too resourceful to simply fall apart altogether. The idea that AI will kill us all in five years is preposterous.
And no, to answer a comment I got on X when I raised these issues, an AI-generated virus is not likely to kill literally all of us, either. When I asked the virologist Dr. Angela Rasmussen about this, she told me “i mean, never say never but the only virus I know that has near total mortality is rabies, and considering it’s not airborne and takes 2-4 weeks to kill you, I don’t think it would take out all of humanity. Ebola doesn’t have 90% mortality like the first outbreak did. It’s closer to 50-60%. Still extremely high but not civilization ending. A pandemic could wipe us out if civilization collapsed...eventually. I’d guess anywhere north of 20-30% mortality would do that. But most pandemic viruses are not that lethal. Even the plague wasn’t that lethal.” In her view, we should be a lot more worried about how AI could cripple civilization than freaking about imaginary AI-created viruses.
§
Which is not at all to say we are home free.
We really need to make a distinction that I have tried to make many times, between catastrophic risk and existential risk. Existential risk has been defined in the literature as the risk of literal extinction; again I think that’s barely above zero, and that AI doesn’t really change that. (Gamma rays could in principle kill us all, too, but I don’t lose sleep over that.)
Catastrophic risk, on the other hand, is quite real. The possibility of nuclear war remain a huge threat to humanity. (I plan to write about that some point; it is much worse than most people realize.)
And AI does in my opinion significantly elevate the risk of catastrophe (say killing 1%+ of humanity or severely damaging modern civilization ), via multiple paths.
AI elevates the risks of bioweapons attacks by making it easier for terrorists to figure out how to develop and implement such attacks.
AI may conceivably elevate the risks that someone will develop a dangerous new virus or other bioweapon.
AI elevates the risks that disinformation will disrupt democracy, by reducing the cost of generating disinformation and increasing the quality (e.g, with ever-better deepfakes)
AI appears to be elevating the risks of serious cyberattacks that could hobble things like banking or electrical grids.
AI introduces a new vector for authoritarians around the globe; by influencing training sets and so forth, authoritarians can influence LLMs in ways that subtly influence their users.
AI also serves as a back door for surveillance, at a scale beyond what Orwell envisioned.
AI is already harming the educational system by leading students into despair and by seducing them into a faux substitute for learning that is easy to use but that ultimately leaves them with poorly developed critical thinking skills.
We should be focusing on things like these, not fairy tales.
Your list of REAL risks at the end is spot on. Every single one of those deserves so much more attention than the sci-fi-superintelligence-takes-over story!
Agree. We should be more worried about the humans behind these companies than the LLMs. These people have the power to destroy more than anything.