To a non techie like me, the inability to separate bullshit from reality is a big problem. It’s hard to determine who bullshits more, the tech itself or its creators. It’s very dangerous.
[Edit: Full disclosure: I should say I do have some experience with programming: I did once make my name endlessly scroll down a TRS-80 during detention.]
I hate to tell you this, but: coming from the clinical neuroscience research side of things, difficulty separating bullshit from reality is a ubiquitous (big) problem. The stakes may differ considerably, of course
No, this is true, especially in contemporary hype-science. I was gonna be historian of science (evolutionary biology) in a former life—not the epistemologically relativist kind, don’t worry!
So it’s so unimaginably powerful at hacking that Anthropic can’t even let independent security researchers inspect it… But Thomas Friedman, a NYT columnist, is allowed to see it.
Makes sense.
After the Claude Code source leak that showed what an incompetent dumpster fire their engineering is, and after all the similar hype announcements about Opus/GPT 5/o3/etc that are now conveniently forgotten, you’ll forgive me for not falling for this one.
I am sure it is good, but not as fear-inspiring as billed. Anthropic has developed a history over the past 12 months of overstating its technology's impact on society. For example, Dario Amodei's dramatic statements on impacts on the job economy. Just another example of building the omnipotence myth. Love Cowork, still hallucinates.
Stop blaming LLMs for the humans behind them. I am pretty sure any human could have coached any of the top-tier LLMs into a similar result. I think Claude is the best, but I can get the others to provide comparable answers.
Humans at Google stopped Project Maven from being a core Google offering because history shows that the US has no scruples in its targeting decisions. Anthropic said to the engineers at Google who organized the labor action that the Google organizers were naive. It turns out the Google organizers were right and everyone is acting like Anthropic had no choice but to collaborate even though we have recent history showing such collaboration can be fought.
I mean, there are many, many (dare I even say most?) humans who would entirely refuse to help kill another human being at all, and certainly at the behest of someone like Trump. To the extent that models can be cajoled or tricked into doing so, yes, that is a failure of the models. Call it a failure of alignment, call it a failure of capabilities, but it is a problem nonetheless.
Scary, ominous and wholly unverifiable nor disprovable claims about the alleged dangers and capabilities of an "emergent" technology? If it smells like tech bros bullshit and desperation, it probably is. Not to say this tech isn't dangerous, but that's true regardless of how intelligent they are. Just another desperate attempt to prop up the AI bubble before it bursts or deflates as it deserves to.
"Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser"
That's a verifiable claim.
"We have reported the above vulnerabilities to the maintainers of the relevant software, and they have all now been patched. For many other vulnerabilities, we are providing a cryptographic hash of the details today (see the Red Team blog), and we will reveal the specifics after a fix is in place."
Another verifiable claim.
And better models are far more dangerous because they are more intelligent. "[Mythos] autonomously found and chained together several vulnerabilities in the Linux kernel—the software that runs most of the world’s servers—to allow an attacker to escalate from ordinary user access to complete control of the machine." That's far worse than a lesser model identifying bugs that could lead to an exploit.
"better models are far more dangerous because they are more intelligent": No LLM is intelligent. They are just faster at guessing possible chains of arguments, which in turn might be fact or fake.
"has already found thousands of high-severity vulnerabilities": I would debate "thousands", but a thousand true positives within an ocean of millions of false positives (which for sure did happen and they are not talking about), this does not sound earth shattering.
But regarding "verifiable claim": let's see what the software's manufacturer will say about this in 3 months time ?
"chained together several vulnerabilities in the Linux kernel" Again, not really impressive given the complexity of the Linux kernel. Also again, let's see what Linus has to say about this ;-). I am looking forward to it.
And there it is, not from Linus but from Greg Kroah-Hartman, long-term Linux kernel maintainer Greg Kroah-Hartman:
"Months ago, we were getting what we called 'AI slop,' AI-generated security reports that were obviously wrong or low quality," he said. "It was kind of funny. It didn't really worry us."
"Things have changed, Kroah-Hartman said. "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
No one is quite sure what's behind it. Asked what changed, Kroah-Hartman was blunt: "We don't know. Nobody seems to know why. Either a lot more tools got a lot better, or people started going, 'Hey, let's start looking at this.'
Note that *nobody* said this has anything to do with Mythos.
But for me the surprise is that a person like Greg says "AI has gotten useful" (paraphrasing) which to me gives a ton of credibility to the usefulness of AI (in coding), more than any Anthropic/OpenAI CEO propaganda ever would.
There is more about the Linux kernel review infrastructure being upgraded with AI:
"Kroah-Hartman credited longtime kernel developer Chris Mason, now at Meta, with pioneering AI-based review workflows. Mason has been running AI review for eBPF and networking for some time. The systemd project is also using the same class of tools for its all-C codebase.
AI reviewers, he stressed, are additive rather than authoritative. "On the review side, it's generating some good reviews. It doesn't get you everything. Some things are still wrong. But it does point out a lot of the obvious things," he said.
OK, I am officially impressed. As a long time Linux user and compile-your-own-kernel fan (a long time ago), I am very much aware of the number of issues in the Linux kernel and the load on the developers to keep things under control.
Without having access to the Mythos software and its data itself, it's very hard to compare it to existing IDS and IPS softwares. Moreover, at that point, why not just publish a list of all the vulnerabilities found so we can actually investigate and corroborate the actual findings for ourselves? Mind you this software is not publicly released yet.
GPT4 was also said to have found vulnerabilities of its own. The hyperbole is very much there, shrouding the real progress and results here, which while impressive does not amount to "untold catastrophe" if unleashed to the public as reported by Anthropic. That's pure fictional BS. THAT'S the unverifiable claims that reek of fearmongering and desperation.
And of course we end up with the pointless circular argument that constantly begs the question.
Anthropic: "We think and know that Mythos is dangerous"
Critics: "then let us analyze and see the model so we can see whether it actually poses a danger and how dangerous it is"
Anthropic: "No we know how dangerous it is but releasing it to the public would be too dangerous. Therefore you must trust us"
There's no winning this argument because by design it's wholly both unverifiable and
can't be disproven the same way we can't prove X number of years into the future that a superintelligence or hell, God, will emerge. Thus, there's little reason to entertain these claims. The findings are worth analyzing and studying, but the claims are little more than marketing meant to fool idiot investors.
"The hyperbole is very much there, shrouding the real progress and results here, which while impressive does not amount to "untold catastrophe" if unleashed to the public as reported by Anthropic. That's pure fictional BS"
1) "Tom Friedman panicked about it in the NYT," aka "Another New York Times columnist, who knows approximately zero about technology, has panicked about AI. Again."
2) Regarding China: what are the odds that China _doesn't_ have a spy working at Anthropic?
Jesus, these grifting AI choads have no courage of conviction - name their machine processor enshitty-gremlins after the old gods…..
Anthropics Ogg-Soggoth, OpenAI Cthulhu’s tentacled spawn, Oracles Yellow King - it’s a fascinating study of self-annihilating dorks inventing a sadomasochistic tech-MackDaddy to make them behave!
I am not surprised that these things may happen (the code pattern finetuning is often quite effective even if it doesn't understand what it is doing), but we should remind ourselves how GPT o3-preview did amazingly well on ARC-AGI-1 (ARC-AGI is a rare useful benchmark): by having been fine-tuned on a lot of the material and by using unlimited compute. Especially the latter is here the big question mark here for me. Not to be dismissed yet as a real risk, there are enough players who can pay for 'unlimited' compute.
Gary, can you point us to where you’ve written very specifically and in detail, about what “government oversight” in this space, means to you? You have been a leading proponent of this message, but I’m at a loss for who in the executive or legislative branches of government could possible oversee or write the rules for the use of AI. I do think that rules that allow it to be more market-driven regulation, specifically the use of liability as the great deterrent (“do big harm, pay big dollars, heads of companies spend some jail time”). It may slow the innovation that so many worry about, but that’s not a bad thing given that we really do want to take some time to understand what is happening, what is actually possible, and now we have enough advancement to see what has actually already happened. The challenge I find, is that those who understand enough about these technologies are not motivated by “government oversight” work. They are do-ers, explorers, one might even call them adventurers (though it’s an adventure many of us would prefer was better thought out ;). But what they’re not, is bureaucrats or regulators wanting to spend their time overseeing how these things advance and get implemented. By putting more responsibility for doing harm, in the hands of those creating these technologies, it will force their hand to be cautious of how far they want to push things. Do they really want something whose results or recommendations are unpredictable dictating the possibility of them going to jail or being stripped of all their wealth? The right set of incentives may help, but there’s still needs to be educated government folks to prosecute these offenses.
Well, levels of safety on food, drinks, cars, planes, machines in general, has been achieved nationally and internationally. It took (a lot of) time, but we got there. Why don't you or we take these existing playbooks to adapt one for AI ?
"use of liability as the great deterrent": That sure would be a great stick to some carrots within a regulation.
"*educated* government folks": Unfortunately currently unavailable with the current administration, but who knows ? Sanity might return, all MAGA supporters come to their senses (fat chance !) or die of measles.
True, in those other disciplines, a way was found for regulating them, i guess it’s possible, it just seems like too many people drinking a heavy dose of kool aid, making rational decisions in how this tech should handled, seemingly difficult.
Absolutely agreed; it’s past time for some real regulation based on evidence of what these frontier models are capable of. Without that we will be continually on the back foot, responding not to what is being released into the world, but rather what was released one or two generations ago, if we’re lucky.
The main problem I have with government regulation of AI - besides my being an anarchist to begin with - is exactly HOW is that going to work.
Oh, we'll have the UN do it. That worked great in preventing the war with Iran.
Then there's the issue of exactly what happens when one country doesn't go along with the ban, secretly. Or someone thinks they have. Iran doesn't have a nuclear weapons program and never did - but that didn't stop the Iran war, either.
Then, considering the explosion of research in the area of LLMs, exactly how will the regulations be written to cover new research? What government agency - national or international - is going to hire who to review every piece of research from every country?
And what about the research done in someone's basement?
It's easy to call for "regulation" - as it is with "gun control".
It's a lot harder to actually DO it. As is the case with gun control, when in the US there are four hundred million firearms, and twenty million concealed carry owners - NOT including criminals.
England has strict gun control.
Now England is considering banning kitchen knives.
So I guess the gun control wasn't enough.
Do I even need to mention drug control? Or even fake pharmaceuticals?
If you want to just control "frontier" AI models from the big companies, that might be made to work. But that means eliminating open source models since anyone can download and host them (if you have the hardware).
But the government will still have access to those frontier models - and use them like Israel used its AI: to identify when an enemy is home so they can kill the whole family.
The US used AI to hit a girl's school in Iran, killing 175 children. The excuse for that was "the school was next to an Iranian navy base." Recently independent journalists went to the school. I saw the video. The "navy base" hadn't been used as such for 18 years. There were weeds growing in the area where the school wasn't located.
The US AI targeting was out of date by eighteen years.
Sorry, but government regulation is not going to save anyone from anything.
Not that I'm worried about this anyway - since LLMs have zero chance of being "AGI". Which is not to say that some future research won't produce something approximating "AGI". Worry about regulating that when it happens.
As for "it doesn't have to be AGI to have risks" - well, that's even harder to regulate, isn't it - a risk you don't even know exists.
How many people thought ChatGPT would cause suicides when it was version 2.0?
Which government agency would have written a regulation to require testing for that?
I'll leave it there. Personally I believe AIs shouldn't even have guardrails or censorship of any kind.
AIs don't kill people. People kill people. And that has never been successfully regulated in human history.
I think I'm more worried about the vulnerabilities it can't find than the ones it can, especially as people start relying on LLMs to tell them their LLM generated code is secure without understanding what it isnt finding. I think expanding exploit generation from a small set of people who often sell to the highest criminal or government-linked hoarder bidder to something Johnny can do on the bus home from school is a net win, there aren't an infinite number of vulnerabilities out there. I was once that kid finding vulnerabilities without AI's help, interested in understanding them and helping companies fix them. It was quite effective in many cases and made executives pay attention and focus resources.
If your local water system or electric company is vulnerable, they are vulnerable regardless of who finds the exploit. Too many companies have been negligently sweeping their risk under the covers of cyber insurance policies instead of investing what is necessary to have proper security architecture and operations. The evolution of the cyber insurance market is going to be very interesting for a lot of reasons.
So, to sum this up, we have a claim from a notoriously shady CEO in a notoriously shady industry. The claim is non-specific and explains close to nothing about the testing conditions. It doesn’t explain what the tests were in any detail, it does not specify what kind of computational resources were used, and it does not give specific numbers. The claim was not put to peer review and no independent testing was allowed. It is even hedged by saying that the chatbot was better than all but the best human beings, which could mean almost anything (being at the 51% percentile is being better than all but the best, for instance, and arguably, the same is true of any number). Despite the supposed extreme danger, the company seems disposed to market the model anyway. The CEO knows that there will be no consequence for deceptive framing or even outright mendacity, because there have not been so far, and that there could be rewards for driving up fear or hype. Many similar exaggerated claims have been made by that company and others in the last few years, including numerous claims of having created AIs that are artificial general intelligence or even are smarter than X human beings put together, without qualification or restriction. Am I missing or incorrect about anything there?
I think it should be clear that we have absolutely no reason to take the claim about the efficacy of the model at spotting cybersecurity flaws at face value! That said, on the off chance that it is true, Dario Amodei should be in prison now, but that has been true of all the AI company leaders for at least the last few years.
I also want to point out that even taking the claim absolutely at face value (don't), it's unclear whether the model really presents the risk claimed. Does spotting exploits better than some or most humans really appreciably increase the risk when there are already humans who can spot those flaws? For that matter, is most cybersecurity risk really due to zero-day exploits? It's far from clear to me that even a bot that spotted exploits better than any human being, or even all possible exploits, would immediately be as dangerous as one might think. I think I would have to find some information that major systems that would be massively life-threatening if compromised tend to have a lot of zero-day flaws allowing dangerous control, and that those flaws were easily spotted by the model but very difficult to spot for human beings, for this to be terribly worrisome. Or perhaps the fear is premised on the notion of every change in concrete chatbot capabilities as an indication of an increase in general capabilities (and malevolence), never mind that we know that not all capabilities are developing at the same rate or are equally “there.”
"Am I missing or incorrect about anything there?" No, aptly summed up.
One could argue that software, compiler and hardware have deterministic rule based behavior, so a machine which knows the myriad of rules better than any human would be superior to any human in spotting errors.
But as anyone who has written even moderately complex software, and worked with "work-in-progress"-compilers and flaky hardware knows that the real bugs are in the unintended dependencies (race conditions, unwanted parallelisms, etc.), *outside* of the rule books.
That means the machine may spot good or bad design patterns which *might* or *might not* lead to problems at run time. So, back to guessing again.
To a non techie like me, the inability to separate bullshit from reality is a big problem. It’s hard to determine who bullshits more, the tech itself or its creators. It’s very dangerous.
[Edit: Full disclosure: I should say I do have some experience with programming: I did once make my name endlessly scroll down a TRS-80 during detention.]
I hate to tell you this, but: coming from the clinical neuroscience research side of things, difficulty separating bullshit from reality is a ubiquitous (big) problem. The stakes may differ considerably, of course
No, this is true, especially in contemporary hype-science. I was gonna be historian of science (evolutionary biology) in a former life—not the epistemologically relativist kind, don’t worry!
One should consider the core arguments and discard the tl;dr.
The core argument here is that AI companies now want a moat to protect their monopoly.
some more informative articles:
https://www.theguardian.com/commentisfree/2023/may/08/ai-machines-hallucinating-naomi-klein
https://davidgerard.co.uk/blockchain/2023/06/03/crypto-collapse-get-in-loser-were-pivoting-to-ai/
https://www.hamiltonnolan.com/p/technology-does-not-solve-political
https://www.wheresyoured.at/burst-damage/
"....Now everything makes perfect sense." What ??
"...Just kidding": Whoa, I really feared for your well being. ;-)
does anyone remember OpenAI saying something similar about GPT2?
https://slate.com/technology/2019/02/openai-gpt2-text-generating-algorithm-ai-dangerous.html
So it’s so unimaginably powerful at hacking that Anthropic can’t even let independent security researchers inspect it… But Thomas Friedman, a NYT columnist, is allowed to see it.
Makes sense.
After the Claude Code source leak that showed what an incompetent dumpster fire their engineering is, and after all the similar hype announcements about Opus/GPT 5/o3/etc that are now conveniently forgotten, you’ll forgive me for not falling for this one.
Things are really getting dicey at the good ship SS "Anthropic", all icebergs and the engines running on fumes. That's what this sounds to me.
I am sure it is good, but not as fear-inspiring as billed. Anthropic has developed a history over the past 12 months of overstating its technology's impact on society. For example, Dario Amodei's dramatic statements on impacts on the job economy. Just another example of building the omnipotence myth. Love Cowork, still hallucinates.
Amodei: "we have to IPO, funds are drying up".
Some VP of something or another: "We have some stuff in a git branch, we don't know yet if it is useful"
Amodei: "hype it up, to ELE and further".
Well Anthropic has proved to be very effective at slaughtering 1000s of Palestinians, Iranians and who knows what other innocents.
Stop blaming LLMs for the humans behind them. I am pretty sure any human could have coached any of the top-tier LLMs into a similar result. I think Claude is the best, but I can get the others to provide comparable answers.
Humans at Google stopped Project Maven from being a core Google offering because history shows that the US has no scruples in its targeting decisions. Anthropic said to the engineers at Google who organized the labor action that the Google organizers were naive. It turns out the Google organizers were right and everyone is acting like Anthropic had no choice but to collaborate even though we have recent history showing such collaboration can be fought.
I mean, there are many, many (dare I even say most?) humans who would entirely refuse to help kill another human being at all, and certainly at the behest of someone like Trump. To the extent that models can be cajoled or tricked into doing so, yes, that is a failure of the models. Call it a failure of alignment, call it a failure of capabilities, but it is a problem nonetheless.
???? Nobody is required to take a contract with the DoD
Don't you get it? I'm not blaming the software! How can anyone blame the software, it's a machine, it's the humans who use it!
no, that would be the 'leaders' of Israel and the US, in that order. with your tax $
We don't know what we want to know until we know what we don't know. Agreed Gary.
To me, the biggest plus is the glasswing defensive stance. Can you imagine OpenAI doing something similar before this was announced? No.
Now, they have to be open or face scrutiny for creating a problem.
The real problem is not just whether Mythos is overhyped or genuinely dangerous.
It’s that the same firms building frontier capability are also narrating its danger, defining its benchmarks, and deciding its release conditions.
That is an unstable governance arrangement.
Scary, ominous and wholly unverifiable nor disprovable claims about the alleged dangers and capabilities of an "emergent" technology? If it smells like tech bros bullshit and desperation, it probably is. Not to say this tech isn't dangerous, but that's true regardless of how intelligent they are. Just another desperate attempt to prop up the AI bubble before it bursts or deflates as it deserves to.
"Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser"
That's a verifiable claim.
"We have reported the above vulnerabilities to the maintainers of the relevant software, and they have all now been patched. For many other vulnerabilities, we are providing a cryptographic hash of the details today (see the Red Team blog), and we will reveal the specifics after a fix is in place."
Another verifiable claim.
And better models are far more dangerous because they are more intelligent. "[Mythos] autonomously found and chained together several vulnerabilities in the Linux kernel—the software that runs most of the world’s servers—to allow an attacker to escalate from ordinary user access to complete control of the machine." That's far worse than a lesser model identifying bugs that could lead to an exploit.
"better models are far more dangerous because they are more intelligent": No LLM is intelligent. They are just faster at guessing possible chains of arguments, which in turn might be fact or fake.
"has already found thousands of high-severity vulnerabilities": I would debate "thousands", but a thousand true positives within an ocean of millions of false positives (which for sure did happen and they are not talking about), this does not sound earth shattering.
But regarding "verifiable claim": let's see what the software's manufacturer will say about this in 3 months time ?
"chained together several vulnerabilities in the Linux kernel" Again, not really impressive given the complexity of the Linux kernel. Also again, let's see what Linus has to say about this ;-). I am looking forward to it.
And there it is, not from Linus but from Greg Kroah-Hartman, long-term Linux kernel maintainer Greg Kroah-Hartman:
"Months ago, we were getting what we called 'AI slop,' AI-generated security reports that were obviously wrong or low quality," he said. "It was kind of funny. It didn't really worry us."
"Things have changed, Kroah-Hartman said. "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
No one is quite sure what's behind it. Asked what changed, Kroah-Hartman was blunt: "We don't know. Nobody seems to know why. Either a lot more tools got a lot better, or people started going, 'Hey, let's start looking at this.'
https://www.theregister.com/2026/03/26/greg_kroahhartman_ai_kernel/
Note that *nobody* said this has anything to do with Mythos.
But for me the surprise is that a person like Greg says "AI has gotten useful" (paraphrasing) which to me gives a ton of credibility to the usefulness of AI (in coding), more than any Anthropic/OpenAI CEO propaganda ever would.
There is more about the Linux kernel review infrastructure being upgraded with AI:
"Kroah-Hartman credited longtime kernel developer Chris Mason, now at Meta, with pioneering AI-based review workflows. Mason has been running AI review for eBPF and networking for some time. The systemd project is also using the same class of tools for its all-C codebase.
AI reviewers, he stressed, are additive rather than authoritative. "On the review side, it's generating some good reviews. It doesn't get you everything. Some things are still wrong. But it does point out a lot of the obvious things," he said.
OK, I am officially impressed. As a long time Linux user and compile-your-own-kernel fan (a long time ago), I am very much aware of the number of issues in the Linux kernel and the load on the developers to keep things under control.
Without having access to the Mythos software and its data itself, it's very hard to compare it to existing IDS and IPS softwares. Moreover, at that point, why not just publish a list of all the vulnerabilities found so we can actually investigate and corroborate the actual findings for ourselves? Mind you this software is not publicly released yet.
GPT4 was also said to have found vulnerabilities of its own. The hyperbole is very much there, shrouding the real progress and results here, which while impressive does not amount to "untold catastrophe" if unleashed to the public as reported by Anthropic. That's pure fictional BS. THAT'S the unverifiable claims that reek of fearmongering and desperation.
And of course we end up with the pointless circular argument that constantly begs the question.
Anthropic: "We think and know that Mythos is dangerous"
Critics: "then let us analyze and see the model so we can see whether it actually poses a danger and how dangerous it is"
Anthropic: "No we know how dangerous it is but releasing it to the public would be too dangerous. Therefore you must trust us"
There's no winning this argument because by design it's wholly both unverifiable and
can't be disproven the same way we can't prove X number of years into the future that a superintelligence or hell, God, will emerge. Thus, there's little reason to entertain these claims. The findings are worth analyzing and studying, but the claims are little more than marketing meant to fool idiot investors.
"The hyperbole is very much there, shrouding the real progress and results here, which while impressive does not amount to "untold catastrophe" if unleashed to the public as reported by Anthropic. That's pure fictional BS"
This Exactly !
1) "Tom Friedman panicked about it in the NYT," aka "Another New York Times columnist, who knows approximately zero about technology, has panicked about AI. Again."
2) Regarding China: what are the odds that China _doesn't_ have a spy working at Anthropic?
Jesus, these grifting AI choads have no courage of conviction - name their machine processor enshitty-gremlins after the old gods…..
Anthropics Ogg-Soggoth, OpenAI Cthulhu’s tentacled spawn, Oracles Yellow King - it’s a fascinating study of self-annihilating dorks inventing a sadomasochistic tech-MackDaddy to make them behave!
I am not surprised that these things may happen (the code pattern finetuning is often quite effective even if it doesn't understand what it is doing), but we should remind ourselves how GPT o3-preview did amazingly well on ARC-AGI-1 (ARC-AGI is a rare useful benchmark): by having been fine-tuned on a lot of the material and by using unlimited compute. Especially the latter is here the big question mark here for me. Not to be dismissed yet as a real risk, there are enough players who can pay for 'unlimited' compute.
Gary, can you point us to where you’ve written very specifically and in detail, about what “government oversight” in this space, means to you? You have been a leading proponent of this message, but I’m at a loss for who in the executive or legislative branches of government could possible oversee or write the rules for the use of AI. I do think that rules that allow it to be more market-driven regulation, specifically the use of liability as the great deterrent (“do big harm, pay big dollars, heads of companies spend some jail time”). It may slow the innovation that so many worry about, but that’s not a bad thing given that we really do want to take some time to understand what is happening, what is actually possible, and now we have enough advancement to see what has actually already happened. The challenge I find, is that those who understand enough about these technologies are not motivated by “government oversight” work. They are do-ers, explorers, one might even call them adventurers (though it’s an adventure many of us would prefer was better thought out ;). But what they’re not, is bureaucrats or regulators wanting to spend their time overseeing how these things advance and get implemented. By putting more responsibility for doing harm, in the hands of those creating these technologies, it will force their hand to be cautious of how far they want to push things. Do they really want something whose results or recommendations are unpredictable dictating the possibility of them going to jail or being stripped of all their wealth? The right set of incentives may help, but there’s still needs to be educated government folks to prosecute these offenses.
Well, levels of safety on food, drinks, cars, planes, machines in general, has been achieved nationally and internationally. It took (a lot of) time, but we got there. Why don't you or we take these existing playbooks to adapt one for AI ?
"use of liability as the great deterrent": That sure would be a great stick to some carrots within a regulation.
"*educated* government folks": Unfortunately currently unavailable with the current administration, but who knows ? Sanity might return, all MAGA supporters come to their senses (fat chance !) or die of measles.
True, in those other disciplines, a way was found for regulating them, i guess it’s possible, it just seems like too many people drinking a heavy dose of kool aid, making rational decisions in how this tech should handled, seemingly difficult.
Absolutely agreed; it’s past time for some real regulation based on evidence of what these frontier models are capable of. Without that we will be continually on the back foot, responding not to what is being released into the world, but rather what was released one or two generations ago, if we’re lucky.
The main problem I have with government regulation of AI - besides my being an anarchist to begin with - is exactly HOW is that going to work.
Oh, we'll have the UN do it. That worked great in preventing the war with Iran.
Then there's the issue of exactly what happens when one country doesn't go along with the ban, secretly. Or someone thinks they have. Iran doesn't have a nuclear weapons program and never did - but that didn't stop the Iran war, either.
Then, considering the explosion of research in the area of LLMs, exactly how will the regulations be written to cover new research? What government agency - national or international - is going to hire who to review every piece of research from every country?
And what about the research done in someone's basement?
It's easy to call for "regulation" - as it is with "gun control".
It's a lot harder to actually DO it. As is the case with gun control, when in the US there are four hundred million firearms, and twenty million concealed carry owners - NOT including criminals.
England has strict gun control.
Now England is considering banning kitchen knives.
So I guess the gun control wasn't enough.
Do I even need to mention drug control? Or even fake pharmaceuticals?
If you want to just control "frontier" AI models from the big companies, that might be made to work. But that means eliminating open source models since anyone can download and host them (if you have the hardware).
But the government will still have access to those frontier models - and use them like Israel used its AI: to identify when an enemy is home so they can kill the whole family.
The US used AI to hit a girl's school in Iran, killing 175 children. The excuse for that was "the school was next to an Iranian navy base." Recently independent journalists went to the school. I saw the video. The "navy base" hadn't been used as such for 18 years. There were weeds growing in the area where the school wasn't located.
The US AI targeting was out of date by eighteen years.
Sorry, but government regulation is not going to save anyone from anything.
Not that I'm worried about this anyway - since LLMs have zero chance of being "AGI". Which is not to say that some future research won't produce something approximating "AGI". Worry about regulating that when it happens.
As for "it doesn't have to be AGI to have risks" - well, that's even harder to regulate, isn't it - a risk you don't even know exists.
How many people thought ChatGPT would cause suicides when it was version 2.0?
Which government agency would have written a regulation to require testing for that?
I'll leave it there. Personally I believe AIs shouldn't even have guardrails or censorship of any kind.
AIs don't kill people. People kill people. And that has never been successfully regulated in human history.
Which is why twenty million Americans carry guns.
I think I'm more worried about the vulnerabilities it can't find than the ones it can, especially as people start relying on LLMs to tell them their LLM generated code is secure without understanding what it isnt finding. I think expanding exploit generation from a small set of people who often sell to the highest criminal or government-linked hoarder bidder to something Johnny can do on the bus home from school is a net win, there aren't an infinite number of vulnerabilities out there. I was once that kid finding vulnerabilities without AI's help, interested in understanding them and helping companies fix them. It was quite effective in many cases and made executives pay attention and focus resources.
If your local water system or electric company is vulnerable, they are vulnerable regardless of who finds the exploit. Too many companies have been negligently sweeping their risk under the covers of cyber insurance policies instead of investing what is necessary to have proper security architecture and operations. The evolution of the cyber insurance market is going to be very interesting for a lot of reasons.
A system doesn’t have to be AGI to carry risks.
Donald J Trump, Kaiser Wilhelm, and Louis XIV bear witness..,.
So, to sum this up, we have a claim from a notoriously shady CEO in a notoriously shady industry. The claim is non-specific and explains close to nothing about the testing conditions. It doesn’t explain what the tests were in any detail, it does not specify what kind of computational resources were used, and it does not give specific numbers. The claim was not put to peer review and no independent testing was allowed. It is even hedged by saying that the chatbot was better than all but the best human beings, which could mean almost anything (being at the 51% percentile is being better than all but the best, for instance, and arguably, the same is true of any number). Despite the supposed extreme danger, the company seems disposed to market the model anyway. The CEO knows that there will be no consequence for deceptive framing or even outright mendacity, because there have not been so far, and that there could be rewards for driving up fear or hype. Many similar exaggerated claims have been made by that company and others in the last few years, including numerous claims of having created AIs that are artificial general intelligence or even are smarter than X human beings put together, without qualification or restriction. Am I missing or incorrect about anything there?
I think it should be clear that we have absolutely no reason to take the claim about the efficacy of the model at spotting cybersecurity flaws at face value! That said, on the off chance that it is true, Dario Amodei should be in prison now, but that has been true of all the AI company leaders for at least the last few years.
I also want to point out that even taking the claim absolutely at face value (don't), it's unclear whether the model really presents the risk claimed. Does spotting exploits better than some or most humans really appreciably increase the risk when there are already humans who can spot those flaws? For that matter, is most cybersecurity risk really due to zero-day exploits? It's far from clear to me that even a bot that spotted exploits better than any human being, or even all possible exploits, would immediately be as dangerous as one might think. I think I would have to find some information that major systems that would be massively life-threatening if compromised tend to have a lot of zero-day flaws allowing dangerous control, and that those flaws were easily spotted by the model but very difficult to spot for human beings, for this to be terribly worrisome. Or perhaps the fear is premised on the notion of every change in concrete chatbot capabilities as an indication of an increase in general capabilities (and malevolence), never mind that we know that not all capabilities are developing at the same rate or are equally “there.”
"Am I missing or incorrect about anything there?" No, aptly summed up.
One could argue that software, compiler and hardware have deterministic rule based behavior, so a machine which knows the myriad of rules better than any human would be superior to any human in spotting errors.
But as anyone who has written even moderately complex software, and worked with "work-in-progress"-compilers and flaky hardware knows that the real bugs are in the unintended dependencies (race conditions, unwanted parallelisms, etc.), *outside* of the rule books.
That means the machine may spot good or bad design patterns which *might* or *might not* lead to problems at run time. So, back to guessing again.