Thank you for posting this very important PSA. As I started reading it, my thought was "surely this only applies to serious developers tempted to use tools that incorporate LLMs." Then I realized that no, it affects everyone, even those who were hoping democratized code applications would be helpful.
JFC! No! While no longer a software developer, I recall how malware was getting into code via altered code libraries, and how to avoid this. The O'Reilly company posts some exploits every month as a warning that malicious exploits and patching are in an arms race. Then there was the hope that AI would help the defenders, but as we know from some recent exploits, this isn't always the case.
What I now fear is that coding might be forced backward in some cases. Code libraries will have to be guaranteed correct before use. AI code tools might have to up their game considerably, or be abandoned, with hand coding and code reviews de rigueur. [Many, many years ago, I met a software developer for a UK military supplier. He told me it was so boring because even a small code change, e.g., a line in C++, had to go through reviews before it could be implemented.]
Funnily enough, Isaac Asimov wrote a short story about how the world was crippled by software/robots deliberately making small mistakes, which upset the functioning of the global economy. More recently, Peter Watts' "Rifters" trilogy described a world with rogue AI software infused everywhere and running rampant through the global networks.
The reality is that code is written by major companies like Microsoft, down through organizations that get ever smaller, to home/retired coders who rely on clean code libraries. I have dabbled with using LLMs to write functions and check that the I/O is correct. I have been dazzled by claims of LLM tools writing complex functional applications via "vibe-coding". enthusiast Ethan Moellick wrote such a post recently, "GPT5 - Just Gets Stuff Done". There is at least one YouTube video of a developer meeting where the presenter states that vibe-coding is the future and why it is superior. If it is, then there had better be far better tools to counter these attacks. It is also making me aware that existing A/V and malware software to protect against malicious code may not be sufficient.
I would hate to have to go back to "stone age" coding, but I fear that the attacks described are going to poison the internet, making it that much more costly to use computers safely.
It isn't superintelligent AI that is the problem, but a proliferation of malicious code generated by AI that will end us, not with a bang, but a whimper.
It's amazing how quickly a modern convenience becomes a fundamental necessity. If we have to manually review every single lib we use and write every line of code in some ancient terminal text editor to avoid the slopification, maybe we'll stop creating bloated ridiculous overcomplicated software when something small and simple would do.
If that means that most people simply won't use computers much or at all anymore, well, what a shame that would be. They'd go back to living like people did in the early 1990s, which I recall was pretty nice but am lately told was "nasty, brutish, and short."
In the middle of the last century, there were people who said the same about telephones and TV (but not radio). Cellphones, especially smartphones (which are computers) have made life a lot easier. I cannot function well or without anxiety if I leave home without one. I rely on my home computer to do all those tasks that once required me to drive everywhere, as well as communicate with people via blogs and comments. If you really think the internet is a bad idea, you are either trolling or have no idea what life is like for many people. The only major downside is that some elderly people who could benefit from using a computer cannot, or will not, do so. This has made their lives unnecessarily restrictive. Life is far easier with teh internet and its services than without it. Yes, our computers are forever in an arms race against malware and viruses, but this is not that different from maliciousness done in the real world, from violence, cons, scams, and general inconveniences that made in teh past also more difficult. Imagine how many lives would not be at risk if Gazans could order up food and water by robot delivery instead of being targets of IDF sharpshooters intent on following their government's genocide orders. I'm sure this applies to other nations, like Ukraine and Haiti.
No, the post is important as it shows our increasing risks of disruption by malicious agents, whether "script kiddies" to government-backed organizations. Sure, the internet makes it easier to inject propaganda or hack infrastructure into enemy nations, but it wasn't as if nations didn't do this with the tools at hand pre-internet.
It is an arms race. Recall when spam was clogging our email? Well that was solved with computers by the platforms. I rarely get a spam email via Gmail that slips past the filters. Viruses? malware? I have A/V and malware software that vigilantly detects and blocks these malicious codes. I have an old Windows 7 computer connected to the internet that has not been infected (AFAIK) despite not having received updates for years. We've known about problem libraries and code that uses these libraries to build applications dependent of libraries for years. AI is increasing the problem as teh post indicates. But it is early days. Forewarned is forearmed. AI will be deployed to counter these attacks, or if unable to do so, AI code-assists will be dropped if the liabilities exceed the supposed productivity gains. Negligence, not AI, has been the cause of some of the big software problems in the last few years. AI might help here, much as lists prevent pilots from making mistakes when checking aircraft status, and doctors from making procedure mistakes.
AI isn't a panacea, and like most technology, a 2-edged sword. What we hope is that its net benefit is positive rather than negative. I believe this will be the case with AI. I don't expect it will either get so good or so disruptive that we will have a Dune-inspired "Butlerian Jihad" against all thinking machines. While I won't live to see it, I could see that a competent home robot would be a huge boon for the disabled and infirm. I suspect Japan will be the first to have them, as its culture appears far less anti-technology than the US.
If you really think the internet is a bad idea, you are either trolling
I was half-joking, but only half at this point.
It’s become increasingly clear that this lackadaisical attitude toward security and now basic sanity is part and parcel to internet culture.
20 years of screaming from the rooftops “if we don’t do something about this we’re going to regret it” has had roughly zero impact. There is little time left to do anything but regret it now.
All the things you are concerned about will come to pass. Nobody will listen.
It almost, for a couple years there, seemed like they might. Legislators were starting to take concerns about “cybersecurity” seriously, whole companies were founded on the principle of putting security and privacy first.
Then they found a new shiny and it all went out the window again. This cycle will continue until the post hoc bandaids and duct tape can’t hold it together anymore.
After which we will have to relearn how to live like it was 1992. Much anxiety and disappointment will ensue. Then we’ll get used to it and it’ll be fine.
Edit:
you should see the “Related Notes” I’m seeing right below this post (viewing it from the notification tab).
Dozens and dozens of nonsense spam posts, half chinese, half english, by something called “Delicia Chan,” with only a couple apparently human notes littered in between. A great illustration of my point. The rot is accelerating and it’s becoming impossible to keep up with it.
Re: email spam, I didn’t see any for years, and now I get 2-3 a day that slip past filters. Google is a the undisputed global leader in machine learning based spam detection. They have almost two decades of expertise and experience at it. If they can’t stop it, nobody can.
You could rewrite your reply, but substitute "global warming," and the experience would be the same.
It isn't internet culture that is to blame, but other forces that are common to other problems that humanity or national leaders fail to deal with adequately. The US won't create good privacy legislation, but it isn't due to some inherent "privacy culture". The same applies to pollution, public health, or any number of issues that are known and have been warned about for decades, with a decided lack of response.
The internet was originally devised as a distributed, fault-tolerant communication system to ensure that a nuclear response was resilient to a communications attack. Then it became a mechanism for institutions to send less critical messages and information. The World Wide Web, which is really what you mean by "internet," was layered on top with a simple interface. Once it was allowed to become a commercial medium, we were off to the races.
AI has been around since the 1960s. What has changed is computing power and the discovery that neural network architecture at scale can mimic several features of human cognition, including speech and "reasoning". As our world gets "Eaten by Software (Andreessen) there are more opportunities for bad actors to prey on us using software, and the complexity increases the attack surfaces. Name another industry where complexity hasn't led to more attempts to prey on it for gain or other reasons.
No, when I say "internet" I mean "internet". The web in particular is a disaster, but no other publicly used protocol is resistant to all the problems inherent in the internet either. Not email in its various forms, not FTP, telnet nor IRC, Gopher or any other ancient zombie protocol (mind you, I like these ancient protocols, but they are what they are). Nor are fancy new ones, like Gemini or IPFS. Nor are bittorrent, bitcoin, urbit nor onion.
Even DNS is vulnerable to spam, scams and slopification, and presently being run as a racket by a cartel of quasi-public corporations.
I wouldn't be surprised if there's some way to use NTP for fraud.
The internet is a fundamentally insecure system at every layer from hardware to application. Permissive systems always are. It makes them flexible and resilient, which was the design intent.
But it also means that bad actors always have the upper hand, and the people who defend against them are always playing catch-up. Thus zero day exploits, not to mention all the other multi-thousand-day exploits still unpatched.
And maybe this was fine, back when it was a few individual malicious actors doing their dirty deeds by hand versus a larger number of good-natured, well-qualified sysadmins.
But now we have botnets and state actors and other people operating at institutional scales, and we do not have institutional responses except for major corporations and governments, who mainly defend their own territory with proprietary solutions that only work at massive scale. The rest of the internet, web and otherwise, has been an absolute wasteland for over a decade. If you don't believe me, try running a web or email server.
And that was a problem *before* you could run an LLM on a commodity GPU trained to generate plausible, human-looking content that could defeat even the institutions' best filters.
I am telling you this is not sustainable and it will not be long now before this thing is all but unusable.
If I were a bad actor, I would wait patiently while the vibe coding trend took off. With each new success, confidence grows in its utility and effectiveness. Usage takes off and more sophisticated offerings are sent to the public with supposed guardrails. Once a reasonable sense of complacency takes over they strike. For a bad actor, you only need one or two successful hits, not durability to make their mark. I am sure the big developers all understand this and are working on prevention but that leaves out the smaller start-ups Public trust, once lost is hard to regain, especially for critical functions and individual private information that compromises their financial safety.
I agree. It is choosing teh optimum time and attack method for what could be a short period of advantage. The problem for attackers is that they are not
monolithic, each perhaps hoping to make a successful attack before the others.
I was heavily involved in computer security from my Air Force days in the 1960s to my retirement from Sandia in 2005, and have followed the field ever since. During that time I watched the tech bros grow rich from the deliberate exploitation of externalities and moral hazard. This latest development suggests that these parasites are nearing the point of killing their host.
Gary, this is just the informed, down-in-the-trenches summary I needed. I have been looking for a way to cover the security aspect of LLMs in an introductory AI course I am piloting this Fall at my institution. It is open to all undergraduates, no requirements on background. I opened up my syllabus on linkedIn asking folks for feedback on what I was missing. One of those undergrads asked me include security! Thank you for this piece.
"But two new technologies are radically increasing what is known as the attack surface (or the space for potential vulnerabilities): LLMs and coding agents."
I call the LLM "The insecure hallucination bus". Everything you attach to the LLM becomes a vulnerability in the chain where the attack surface is essentially anything that can be expressed in human language.
There are no solutions except as you state to simply not use it. The LLM has to be treated as an untrusted hostile machine, but of course that kills most use cases as it makes LLM utilization often impractical.
Morris Worm exploited a known bug in the C gets() instruction in sendmail that the Unix people couldn't be arsed to fix. I know they knew of the problem because fellow programmer sent them a warning in the spring of '88.
I don't think the enormity of business code using GitHub repos without an understanding of the underlying code can be overstated. Thank goodness for companies using technology to detect these situations.
Great post very important. Thanks for sharing. One thing: lots of references to developers being in a hurry. There’s no hope for individual independent developers. However, otherwise that’s a responsibility of development managers and organizational executives. In any case negative feedback is going to limit consequences, though likely not before billions burn.
Gary, many in the mainstream characterize you, unfairly in my opinion, as “anti-AI.” Aside from your well-founded critique of the dominant paradigm, you seem to think AI, in general, is a worthwhile technology. Why do you think AI is an important technology for the human species to pursue? What is your case for the societal importance of a properly developed AI? What would you say to someone who asserts that we should stop investing in all forms of AI (because there are people who think this)?
My quibble with 'better people' is that that group is larger than the tech companies who develop AI; it includes their customers. It's the governments and businesses that are easily seduced by the potential cost savings of automation using general purpose AI.
I am skeptical that the power dynamics of that kind of AI will improve, even if AI itself becomes more reliable. This is where Karen Hao's point about more task-specific models built for real problems comes into play. Even then, it'll be tricky.
And maybe we can substitute better organizations for better people (or not, because this doesn’t work very often, but one can dream). If every software development organization had a team responsible for importing and inspecting all external packages and libraries for exploits, and for reviewing LLM inputs for other attacks, the attack surface of the development process could at least be significantly reduced.
This of course would add time and cost to the development process, so it’s unlikely to be done in the current business environment.
Funded by the government certainly, done by, I’m not so sure. I think the, let’s call it the “library security team”, needs to be close to the development team to ensure good communication between them. The library team should probably participate in code reviews and be involved in developing tests that go into the main test repository. That seems like a relationship that an external goverment team would find difficult because of proprietary secrets and other sensitive project information.
Truly illuminating. On the other hand and as the authors understand well is going to be impossible to stop the tide. To take my own example, I am a researcher in physics with 40+ years programming experience. I have “resisted” agents until recently. I use now Claude code, although restricted to coding tasks in specific repos . The boost in productivity is huge. It follows that people will choose productivity instead of safety even if that implies risks… at least until disaster strikes. The question is whether there are solutions to be implemented to mitigate risk. Local models? Restricted access/ actions? I would like very much to read more on those recommendations
From a year ago, I wonder whether this is still true:
"In a bold move addressing some major cybersecurity concerns that have plagued the company in recent months, Microsoft has linked executive compensation to the company’s security performance."
Gary, I encourage you to check out a company called Verses AI. Their Active Interference technology doesn't use a large language model, and it is faster and more accurate than all the other AI competitors out there. I'm stunned at how little media coverage the company has received.
I wonder whether using LLM’s for coding will result in more bugs in general, not just in security. There have been reports lately that the Google nest product has become almost unusable due to software quality issues. It’s shocking to me that this would happen to a product from Google. I wonder if there is a connection here to the claim I’ve heard that Google produces 80% of their code with LLM assistance.
My experience with copilot is more bugs. And difficult to spot ones. Also I see less code reuse. This is anecdotal I run a small dev team, but right now I'm on the fence that there is overall productivity gain. Don't let Jr Developers use these tools without good oversight (but that's typically true even without these tools)
Thank you for posting this very important PSA. As I started reading it, my thought was "surely this only applies to serious developers tempted to use tools that incorporate LLMs." Then I realized that no, it affects everyone, even those who were hoping democratized code applications would be helpful.
JFC! No! While no longer a software developer, I recall how malware was getting into code via altered code libraries, and how to avoid this. The O'Reilly company posts some exploits every month as a warning that malicious exploits and patching are in an arms race. Then there was the hope that AI would help the defenders, but as we know from some recent exploits, this isn't always the case.
What I now fear is that coding might be forced backward in some cases. Code libraries will have to be guaranteed correct before use. AI code tools might have to up their game considerably, or be abandoned, with hand coding and code reviews de rigueur. [Many, many years ago, I met a software developer for a UK military supplier. He told me it was so boring because even a small code change, e.g., a line in C++, had to go through reviews before it could be implemented.]
Funnily enough, Isaac Asimov wrote a short story about how the world was crippled by software/robots deliberately making small mistakes, which upset the functioning of the global economy. More recently, Peter Watts' "Rifters" trilogy described a world with rogue AI software infused everywhere and running rampant through the global networks.
The reality is that code is written by major companies like Microsoft, down through organizations that get ever smaller, to home/retired coders who rely on clean code libraries. I have dabbled with using LLMs to write functions and check that the I/O is correct. I have been dazzled by claims of LLM tools writing complex functional applications via "vibe-coding". enthusiast Ethan Moellick wrote such a post recently, "GPT5 - Just Gets Stuff Done". There is at least one YouTube video of a developer meeting where the presenter states that vibe-coding is the future and why it is superior. If it is, then there had better be far better tools to counter these attacks. It is also making me aware that existing A/V and malware software to protect against malicious code may not be sufficient.
I would hate to have to go back to "stone age" coding, but I fear that the attacks described are going to poison the internet, making it that much more costly to use computers safely.
It isn't superintelligent AI that is the problem, but a proliferation of malicious code generated by AI that will end us, not with a bang, but a whimper.
It's amazing how quickly a modern convenience becomes a fundamental necessity. If we have to manually review every single lib we use and write every line of code in some ancient terminal text editor to avoid the slopification, maybe we'll stop creating bloated ridiculous overcomplicated software when something small and simple would do.
If that means that most people simply won't use computers much or at all anymore, well, what a shame that would be. They'd go back to living like people did in the early 1990s, which I recall was pretty nice but am lately told was "nasty, brutish, and short."
This is fine. The internet was a mistake anyway.
In the middle of the last century, there were people who said the same about telephones and TV (but not radio). Cellphones, especially smartphones (which are computers) have made life a lot easier. I cannot function well or without anxiety if I leave home without one. I rely on my home computer to do all those tasks that once required me to drive everywhere, as well as communicate with people via blogs and comments. If you really think the internet is a bad idea, you are either trolling or have no idea what life is like for many people. The only major downside is that some elderly people who could benefit from using a computer cannot, or will not, do so. This has made their lives unnecessarily restrictive. Life is far easier with teh internet and its services than without it. Yes, our computers are forever in an arms race against malware and viruses, but this is not that different from maliciousness done in the real world, from violence, cons, scams, and general inconveniences that made in teh past also more difficult. Imagine how many lives would not be at risk if Gazans could order up food and water by robot delivery instead of being targets of IDF sharpshooters intent on following their government's genocide orders. I'm sure this applies to other nations, like Ukraine and Haiti.
No, the post is important as it shows our increasing risks of disruption by malicious agents, whether "script kiddies" to government-backed organizations. Sure, the internet makes it easier to inject propaganda or hack infrastructure into enemy nations, but it wasn't as if nations didn't do this with the tools at hand pre-internet.
It is an arms race. Recall when spam was clogging our email? Well that was solved with computers by the platforms. I rarely get a spam email via Gmail that slips past the filters. Viruses? malware? I have A/V and malware software that vigilantly detects and blocks these malicious codes. I have an old Windows 7 computer connected to the internet that has not been infected (AFAIK) despite not having received updates for years. We've known about problem libraries and code that uses these libraries to build applications dependent of libraries for years. AI is increasing the problem as teh post indicates. But it is early days. Forewarned is forearmed. AI will be deployed to counter these attacks, or if unable to do so, AI code-assists will be dropped if the liabilities exceed the supposed productivity gains. Negligence, not AI, has been the cause of some of the big software problems in the last few years. AI might help here, much as lists prevent pilots from making mistakes when checking aircraft status, and doctors from making procedure mistakes.
AI isn't a panacea, and like most technology, a 2-edged sword. What we hope is that its net benefit is positive rather than negative. I believe this will be the case with AI. I don't expect it will either get so good or so disruptive that we will have a Dune-inspired "Butlerian Jihad" against all thinking machines. While I won't live to see it, I could see that a competent home robot would be a huge boon for the disabled and infirm. I suspect Japan will be the first to have them, as its culture appears far less anti-technology than the US.
If you really think the internet is a bad idea, you are either trolling
I was half-joking, but only half at this point.
It’s become increasingly clear that this lackadaisical attitude toward security and now basic sanity is part and parcel to internet culture.
20 years of screaming from the rooftops “if we don’t do something about this we’re going to regret it” has had roughly zero impact. There is little time left to do anything but regret it now.
All the things you are concerned about will come to pass. Nobody will listen.
It almost, for a couple years there, seemed like they might. Legislators were starting to take concerns about “cybersecurity” seriously, whole companies were founded on the principle of putting security and privacy first.
Then they found a new shiny and it all went out the window again. This cycle will continue until the post hoc bandaids and duct tape can’t hold it together anymore.
After which we will have to relearn how to live like it was 1992. Much anxiety and disappointment will ensue. Then we’ll get used to it and it’ll be fine.
Edit:
you should see the “Related Notes” I’m seeing right below this post (viewing it from the notification tab).
Dozens and dozens of nonsense spam posts, half chinese, half english, by something called “Delicia Chan,” with only a couple apparently human notes littered in between. A great illustration of my point. The rot is accelerating and it’s becoming impossible to keep up with it.
Re: email spam, I didn’t see any for years, and now I get 2-3 a day that slip past filters. Google is a the undisputed global leader in machine learning based spam detection. They have almost two decades of expertise and experience at it. If they can’t stop it, nobody can.
You could rewrite your reply, but substitute "global warming," and the experience would be the same.
It isn't internet culture that is to blame, but other forces that are common to other problems that humanity or national leaders fail to deal with adequately. The US won't create good privacy legislation, but it isn't due to some inherent "privacy culture". The same applies to pollution, public health, or any number of issues that are known and have been warned about for decades, with a decided lack of response.
The internet was originally devised as a distributed, fault-tolerant communication system to ensure that a nuclear response was resilient to a communications attack. Then it became a mechanism for institutions to send less critical messages and information. The World Wide Web, which is really what you mean by "internet," was layered on top with a simple interface. Once it was allowed to become a commercial medium, we were off to the races.
AI has been around since the 1960s. What has changed is computing power and the discovery that neural network architecture at scale can mimic several features of human cognition, including speech and "reasoning". As our world gets "Eaten by Software (Andreessen) there are more opportunities for bad actors to prey on us using software, and the complexity increases the attack surfaces. Name another industry where complexity hasn't led to more attempts to prey on it for gain or other reasons.
No, when I say "internet" I mean "internet". The web in particular is a disaster, but no other publicly used protocol is resistant to all the problems inherent in the internet either. Not email in its various forms, not FTP, telnet nor IRC, Gopher or any other ancient zombie protocol (mind you, I like these ancient protocols, but they are what they are). Nor are fancy new ones, like Gemini or IPFS. Nor are bittorrent, bitcoin, urbit nor onion.
Even DNS is vulnerable to spam, scams and slopification, and presently being run as a racket by a cartel of quasi-public corporations.
I wouldn't be surprised if there's some way to use NTP for fraud.
The internet is a fundamentally insecure system at every layer from hardware to application. Permissive systems always are. It makes them flexible and resilient, which was the design intent.
But it also means that bad actors always have the upper hand, and the people who defend against them are always playing catch-up. Thus zero day exploits, not to mention all the other multi-thousand-day exploits still unpatched.
And maybe this was fine, back when it was a few individual malicious actors doing their dirty deeds by hand versus a larger number of good-natured, well-qualified sysadmins.
But now we have botnets and state actors and other people operating at institutional scales, and we do not have institutional responses except for major corporations and governments, who mainly defend their own territory with proprietary solutions that only work at massive scale. The rest of the internet, web and otherwise, has been an absolute wasteland for over a decade. If you don't believe me, try running a web or email server.
And that was a problem *before* you could run an LLM on a commodity GPU trained to generate plausible, human-looking content that could defeat even the institutions' best filters.
I am telling you this is not sustainable and it will not be long now before this thing is all but unusable.
If I were a bad actor, I would wait patiently while the vibe coding trend took off. With each new success, confidence grows in its utility and effectiveness. Usage takes off and more sophisticated offerings are sent to the public with supposed guardrails. Once a reasonable sense of complacency takes over they strike. For a bad actor, you only need one or two successful hits, not durability to make their mark. I am sure the big developers all understand this and are working on prevention but that leaves out the smaller start-ups Public trust, once lost is hard to regain, especially for critical functions and individual private information that compromises their financial safety.
I agree. It is choosing teh optimum time and attack method for what could be a short period of advantage. The problem for attackers is that they are not
monolithic, each perhaps hoping to make a successful attack before the others.
Maybe AIs can help with that game, too? ;-(
Tit for Tat.
I was heavily involved in computer security from my Air Force days in the 1960s to my retirement from Sandia in 2005, and have followed the field ever since. During that time I watched the tech bros grow rich from the deliberate exploitation of externalities and moral hazard. This latest development suggests that these parasites are nearing the point of killing their host.
Gary, this is just the informed, down-in-the-trenches summary I needed. I have been looking for a way to cover the security aspect of LLMs in an introductory AI course I am piloting this Fall at my institution. It is open to all undergraduates, no requirements on background. I opened up my syllabus on linkedIn asking folks for feedback on what I was missing. One of those undergrads asked me include security! Thank you for this piece.
"But two new technologies are radically increasing what is known as the attack surface (or the space for potential vulnerabilities): LLMs and coding agents."
I call the LLM "The insecure hallucination bus". Everything you attach to the LLM becomes a vulnerability in the chain where the attack surface is essentially anything that can be expressed in human language.
There are no solutions except as you state to simply not use it. The LLM has to be treated as an untrusted hostile machine, but of course that kills most use cases as it makes LLM utilization often impractical.
Utterly terrifying. Thanks we needed that.
Morris Worm exploited a known bug in the C gets() instruction in sendmail that the Unix people couldn't be arsed to fix. I know they knew of the problem because fellow programmer sent them a warning in the spring of '88.
I don't think the enormity of business code using GitHub repos without an understanding of the underlying code can be overstated. Thank goodness for companies using technology to detect these situations.
Great post very important. Thanks for sharing. One thing: lots of references to developers being in a hurry. There’s no hope for individual independent developers. However, otherwise that’s a responsibility of development managers and organizational executives. In any case negative feedback is going to limit consequences, though likely not before billions burn.
Thanks, Gary.
Gary, many in the mainstream characterize you, unfairly in my opinion, as “anti-AI.” Aside from your well-founded critique of the dominant paradigm, you seem to think AI, in general, is a worthwhile technology. Why do you think AI is an important technology for the human species to pursue? What is your case for the societal importance of a properly developed AI? What would you say to someone who asserts that we should stop investing in all forms of AI (because there are people who think this)?
If done right, by better people, i think it could hugely advance science, medicine, and education
My quibble with 'better people' is that that group is larger than the tech companies who develop AI; it includes their customers. It's the governments and businesses that are easily seduced by the potential cost savings of automation using general purpose AI.
I am skeptical that the power dynamics of that kind of AI will improve, even if AI itself becomes more reliable. This is where Karen Hao's point about more task-specific models built for real problems comes into play. Even then, it'll be tricky.
And maybe we can substitute better organizations for better people (or not, because this doesn’t work very often, but one can dream). If every software development organization had a team responsible for importing and inspecting all external packages and libraries for exploits, and for reviewing LLM inputs for other attacks, the attack surface of the development process could at least be significantly reduced.
This of course would add time and cost to the development process, so it’s unlikely to be done in the current business environment.
This suggests that it is a public good and should be done/funded by the government, not privately.
Do you apply this reasoning to AGI, as well?
Funded by the government certainly, done by, I’m not so sure. I think the, let’s call it the “library security team”, needs to be close to the development team to ensure good communication between them. The library team should probably participate in code reviews and be involved in developing tests that go into the main test repository. That seems like a relationship that an external goverment team would find difficult because of proprietary secrets and other sensitive project information.
Truly illuminating. On the other hand and as the authors understand well is going to be impossible to stop the tide. To take my own example, I am a researcher in physics with 40+ years programming experience. I have “resisted” agents until recently. I use now Claude code, although restricted to coding tasks in specific repos . The boost in productivity is huge. It follows that people will choose productivity instead of safety even if that implies risks… at least until disaster strikes. The question is whether there are solutions to be implemented to mitigate risk. Local models? Restricted access/ actions? I would like very much to read more on those recommendations
From a year ago, I wonder whether this is still true:
"In a bold move addressing some major cybersecurity concerns that have plagued the company in recent months, Microsoft has linked executive compensation to the company’s security performance."
Lying awake thinking of nuclear power facilities around the world and hoping someone is dealing with this problem
Gary, I encourage you to check out a company called Verses AI. Their Active Interference technology doesn't use a large language model, and it is faster and more accurate than all the other AI competitors out there. I'm stunned at how little media coverage the company has received.
i know about them and will cover eventually
Great! I believe Verses AI is the biggest business story of the year, if not the century!
"How We Exploited CodeRabbit: From a Simple PR to RCE and Write Access on 1M Repositories"
was posted today by Nils/Nathan
https://research.kudelskisecurity.com/2025/08/19/how-we-exploited-coderabbit-from-a-simple-pr-to-rce-and-write-access-on-1m-repositories/
I wonder whether using LLM’s for coding will result in more bugs in general, not just in security. There have been reports lately that the Google nest product has become almost unusable due to software quality issues. It’s shocking to me that this would happen to a product from Google. I wonder if there is a connection here to the claim I’ve heard that Google produces 80% of their code with LLM assistance.
My experience with copilot is more bugs. And difficult to spot ones. Also I see less code reuse. This is anecdotal I run a small dev team, but right now I'm on the fence that there is overall productivity gain. Don't let Jr Developers use these tools without good oversight (but that's typically true even without these tools)