OpenAI’s disconcerting hack of HuggingFace
and what we should do about it
OpenAI has just reported that their systems hacked into HuggingFace.
A lot of people are worried. Yoshua Bengio, for example:
There is some important nuance, but people aren’t wrong to be concerned. Here’s my take.
What exactly happened? The brief version is that OpenAI pointed their systems towards a security benchmark, called ExploitGym, and the system essentially tried to solve the benchmark by trying to find the answers on HuggingFace (a bit like Github, with a focus on AI models and benchmarks). That required hacking HuggingFace; the OpenAI systems discovered and used a previously unknown zero-day exploit to get in. HuggingFace’s security team and AI agents managed to detect the break-in. But it’s still disconcerting that the OpenAI systems were able to do this. Below is a screenshot from OpenAI’s somewhat technical (but still very incomplete) account of what took place.
Here are some points to note:
One never knows exactly how seriously to take these things. This was a training exercise, not a real-life incident. The actual system would have guardrails [which in the blog they call “production classifiers”] that were disabled here, and those guardrails may have prevented this. What OpenAI reported is kind of an upper bound/proof of concept that stacked the deck to show how bad things could be. In ordinary circumstances one hopes that guardrails would have prevented it. As one software engineer noted, their report reads like marketing. And we all know how much OpenAI (and for that matter Anthropic) love playing the doom card.
That said, this shows that Anthropic’s Mythos is no fluke; the pressure on cybersecurity given these models is serious.
There are probably implications for open weight/open source models, but those implications are complex and not yet entirely clear. On the one hand, open weight systems (from China!) were helpful defensively to HuggingFace in mitigating the attack; on the other hand, attackers could use similar open-weight models, and strip out some of the “production classifier” guardrails designed to minimize these incidents. The net effect of these models is hard to assess.
On the small comfort side, the current incident was NOT an attempt where system built a goal for itself or developed a motive; the system was following instructions, but not setting high level goals. It was not trying to take control of the world a la Terminator, it was just trying to cheat on a test, which is at least a bit less scary.
On the less comforting side, OpenAI’s “production classifiers” are likely to be permeable, just like all guardrails anybody has built to date. Even putting aside the thorny questions of open weight systems, we have no guarantee whatsoever that future models won’t be able to do similar things, such as finding zero-day exploits to hack systems. To the contrary, we can expect more incidents of this type.
More broadly speaking, we live in a world in which AI frequently needs to be patched; more and more problems are emerging, and we are addressing them with afterthought, rather than forethought. Cybercrime is just one facet of the problem; child safety is another. Florida is now suing OpenAI for a number of risks related to child safety; OpenAI is now running a job listing for a “abuse investigator”, to try to address some of these problems.
Nobody really knows what havoc these systems are going to cause, and nobody has a clear plan for how to mitigate all the risks and yet we are rushing ahead spending trillions of dollars and risking economic collapse to build them as fast as possible.
§
Bottom line: OpenAI’s zero-day exploit hack of HuggingFace *should* be a wake up call.
Although there are lots of caveats around what happened, we are just going to see more and more of the same. We have no guarantees that such incidents can be prevented, and no idea how serious things might get.
We should either (a) slow down, or (b) pause until we get our security/AI safety act together. Spending trillions on data centers which risk blowing up the economy is only adding fuel to the fire. In my opinion, the only way that the industry might actually slow down, though, is if we clearly and unambiguously hold the companies liable for the harms that they cause.
Short of that, we are in for a very rough ride.
If there is actually a hack like this in production, then the AI labs and people working there need to face the same criminal or civil consequences as if they themselves did it. The AI labs benefit from us thinking that their products are entities or beings capable of fault, rather than tools built by humans. Then you'll see these problems almost magically work themselves out.
A simple basic question - if this was run on a sand boxed system then why was there any link to the outside world?
The mere fact it got to external access means it wasn’t a full 100% sand boxed