Excellent post. The risks lying in wait for us at the intersection of AI and Cybersecurity are much bigger than people think. You may find my article on how to approach this interesting:
Oh but no worries, governments will do their job and regulate, particularly if Thiel/Andreessen come to power (ooops, I meant Trump/Vance). Nothing to fear but fear itself. ;P
The people of planet have to realize we have given our power/sovereign selves to the Digerati under the guise of productivity/efficiency and convenience for ourselves. Folks who believe in the false concepts of utopia and perfection are surprised at these events..... I am not. This will be our future at times. Can't turn back now right? These software types have sold a false future. Strap up!!! When you give up your power this is what happens. However, I am not surprised by this. Man is not God or nature. We are imperfect which most do not embrace
I woke up. This morning. To a global IT outage. Not here, we're good (knock on... wood? certainly not software lol) And thought, "Lovely. A single company can take down the operations of an entire planet."
And it wasn't intentional.
Egyptian pyramids everywhere all at once.
If this doesn't wake up regulatory bodies nothing will.
"Move fast and break things" sounds great until you actually break things. But 'move fast' is not what happened here.
What we are seeing is a logical consequence of large logical landscapes being brittle, because (discrete) logic is unforgiving. One bit wrong can bring a system down, and we address this — ironically — by adding more logic. This is for instance also the reason why change becomes ever harder (and slower).
The IT-revolution has in part been an exchange between increasing productivity but also increasing brittleness (and by that decreasing agility — for organizations and society alike). We're not approaching a 'singularity point', we're approaching a 'complexity crunch'. AI — being huge systems — is not going to solve this, it will simply add to the effect.
I would suggest two better metaphors than my old pyramid one (a) trying to build a livable city of playing cards (b) trying to scale up a doghouse many orders of magnitude.
A fun blast from the past was that the Apollo Guidance Computer Software (OS by Hal Laning, rest of SW effort was led by Margaret Hamilton), was writen in low level assembly code, there was a lot of it, and it was very difficult to debug. https://en.wikipedia.org/wiki/Apollo_Guidance_Computer
However, both Hal and Margaret believed in "before the fact" engineering (Margaret's phrase), meaning: make sure the software can deal with problems successfully to avoid fatal errors. This saved the Apollo 11 landing. Perhaps more astounding is that there were no SW failures over all of the Apollo missions.
When asked what they were doing, she said "Software Engineering" (and, they were).
I'm pretty sure that most computerists today do not really know what the term "Engineering" is supposed to mean, and that most have never done any kind of actual engineering of any kind.
"Gary Marcus is deeply distressed that certain tech leaders and investors are putting massive support behind the presidential candidate least likely to regulate software."
A crucial point, although the tech industry will attempt to buy anyone in power. But we know that one side of our political equation doesn't believe in regulation to begin with. Might makes right when it's a free-for-all. Things could get pretty damn good for less than 1% of us. Not so good for the rest of us.
As someone that used to do coding / programming / software development, I am all too familiar with how one missing or extra semi-colon; can bring an enormous system to a crashing halt. I don't think folks who haven't messed with computers or software at a low level realize that.
The fact that now it may be some module with thousands or millions of lines of code, or a router, or a cache chip, whatever, makes no difference – it's just a matter of a changed relative scale. And now we have these opaque, black-box, ultra complex LLM and "AI" systems with massive databases.
Anyway, that whole experience, plus being a (former) computer consultant who helped people who had lost their PhD thesis or whatever to a hard drive crash (and I lost years of photos when I dropped an external laptop drive) know how damn brittle these machines and systems are.
Also, as a wise uncle said long ago, when he the young me devoting my time to this computer stuff: "What happens when the power goes off?"
Can anyone say "live a simple, independent, free life"? :)
By the way, I am not advocating a rejection of technology at all, far from it. I love tech. Rather see what you have faith in, and where your interest and attention go, and how that plays out. ;;; and BACK UP! :)
The reason is due to the lousy software called CrowdStrike.
This so-called CrowdStrike is a security protection software, which is said to be able to prevent ransomware and the like, but it is extremely difficult to use. On the Windows system, even without any operation, the CPU usage rate is around 20%, which is worse than Kaspersky was back in the day. It imposes many restrictions on your system and is very unfriendly to software development.
For example, if you casually write a Windows program with Visual Studio, once compiled, it is directly killed, and even some Python scripts, if they contain download functions, are considered viruses, which is the type that would rather kill a thousand innocents than let one go.
In terms of functionality, it is far inferior to MSE. This kind of thing is a disaster when used in software development companies.
Combined with the company's domain management and domain policies, this thing cannot be turned off, and it forces updates, and even forces you to restart for updates.
And I don't know how this company does its public relations, now European and American companies, especially American companies, have it installed as a standard, and this company's market value is also very high (80 billion US dollars). I seriously doubt whether this company's market value matches its technical capabilities, and I even suspect that it has achieved such a large market through commercial bribery.
I think you're quite wrong...the pyramids were designed by geniuses to have extraordinary structural integrity. Are you suggesting that Microsoft Windows will last 2000 years?
You are right that the pyramids (particularly ones like the Great Pyramid at Giza) are a very poor analogy for brittle, illogical, error riddled software.
In fact, one would be hard pressed to pick a worse analogy. I bet even ChatGPT could tell us that much.
And MS Windows lasting 2000 years?
Ha ha ha . It’s very lucky to last 2000 minutes before it crashes.
Although Windows as a burial chamber (The Blue screen of Death) IS in keeping with the pyramid analogy.
My wife, at O'Hare airport, said that the Arrivals / Departures boards were all showing the blue screen with a frowny-face and a message like "Unrecoverable error." I have a sci-fi story somewhere in which the characters travel to a distant planet and find robots that seem eerily familiar...until they take one apart and find MS Windows patches in its OS.
Thanks for this. A fundamental problem is that the software industry has no real incentive to clean up their act. Last year we saw a repeat of the old financial industry two step in which they wring their hands before a Congressional hearing about how they need regulation while simultaneously lobbying for none. And our legislators are nearly clueless when it comes to anything technical. Their approach is to associate expertise with corporate stature, when in fact most of the suits are motivated only by their desire for money and power.
Thank you, have been waiting for the coverage to point this out. And whether a
GenAI code generator was involved in this seems a reasonable question, given that it reduces attention to detail….
Excellent post. The risks lying in wait for us at the intersection of AI and Cybersecurity are much bigger than people think. You may find my article on how to approach this interesting:
https://open.substack.com/pub/aipdp/p/openatom-5-national-ai-and-cybersafety?utm_campaign=post&utm_medium=web
Oh but no worries, governments will do their job and regulate, particularly if Thiel/Andreessen come to power (ooops, I meant Trump/Vance). Nothing to fear but fear itself. ;P
Right, as if the technocrats don’t already own Washington and Sacramento.
The people of planet have to realize we have given our power/sovereign selves to the Digerati under the guise of productivity/efficiency and convenience for ourselves. Folks who believe in the false concepts of utopia and perfection are surprised at these events..... I am not. This will be our future at times. Can't turn back now right? These software types have sold a false future. Strap up!!! When you give up your power this is what happens. However, I am not surprised by this. Man is not God or nature. We are imperfect which most do not embrace
I woke up. This morning. To a global IT outage. Not here, we're good (knock on... wood? certainly not software lol) And thought, "Lovely. A single company can take down the operations of an entire planet."
And it wasn't intentional.
Egyptian pyramids everywhere all at once.
If this doesn't wake up regulatory bodies nothing will.
Gonna be That Guy: the pyramids weren't built by slaves. The builders were laborers, paid for their time and work.
If only the same could be said of modern LLMs.
I was gonna be that one too.
Well, that's very nice to know, actually :)
Glad you pointed that out. I get tired of reading about Egyptian slaves building the pyramids.
https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/
Good post.
"Move fast and break things" sounds great until you actually break things. But 'move fast' is not what happened here.
What we are seeing is a logical consequence of large logical landscapes being brittle, because (discrete) logic is unforgiving. One bit wrong can bring a system down, and we address this — ironically — by adding more logic. This is for instance also the reason why change becomes ever harder (and slower).
The IT-revolution has in part been an exchange between increasing productivity but also increasing brittleness (and by that decreasing agility — for organizations and society alike). We're not approaching a 'singularity point', we're approaching a 'complexity crunch'. AI — being huge systems — is not going to solve this, it will simply add to the effect.
See https://ea.rna.nl/all-that-it-what-is-it-doing-to-us/
Hi Gary
I would suggest two better metaphors than my old pyramid one (a) trying to build a livable city of playing cards (b) trying to scale up a doghouse many orders of magnitude.
A fun blast from the past was that the Apollo Guidance Computer Software (OS by Hal Laning, rest of SW effort was led by Margaret Hamilton), was writen in low level assembly code, there was a lot of it, and it was very difficult to debug. https://en.wikipedia.org/wiki/Apollo_Guidance_Computer
However, both Hal and Margaret believed in "before the fact" engineering (Margaret's phrase), meaning: make sure the software can deal with problems successfully to avoid fatal errors. This saved the Apollo 11 landing. Perhaps more astounding is that there were no SW failures over all of the Apollo missions.
When asked what they were doing, she said "Software Engineering" (and, they were).
I'm pretty sure that most computerists today do not really know what the term "Engineering" is supposed to mean, and that most have never done any kind of actual engineering of any kind.
"Gary Marcus is deeply distressed that certain tech leaders and investors are putting massive support behind the presidential candidate least likely to regulate software."
A crucial point, although the tech industry will attempt to buy anyone in power. But we know that one side of our political equation doesn't believe in regulation to begin with. Might makes right when it's a free-for-all. Things could get pretty damn good for less than 1% of us. Not so good for the rest of us.
As someone that used to do coding / programming / software development, I am all too familiar with how one missing or extra semi-colon; can bring an enormous system to a crashing halt. I don't think folks who haven't messed with computers or software at a low level realize that.
The fact that now it may be some module with thousands or millions of lines of code, or a router, or a cache chip, whatever, makes no difference – it's just a matter of a changed relative scale. And now we have these opaque, black-box, ultra complex LLM and "AI" systems with massive databases.
Anyway, that whole experience, plus being a (former) computer consultant who helped people who had lost their PhD thesis or whatever to a hard drive crash (and I lost years of photos when I dropped an external laptop drive) know how damn brittle these machines and systems are.
Also, as a wise uncle said long ago, when he the young me devoting my time to this computer stuff: "What happens when the power goes off?"
Can anyone say "live a simple, independent, free life"? :)
By the way, I am not advocating a rejection of technology at all, far from it. I love tech. Rather see what you have faith in, and where your interest and attention go, and how that plays out. ;;; and BACK UP! :)
The reason is due to the lousy software called CrowdStrike.
This so-called CrowdStrike is a security protection software, which is said to be able to prevent ransomware and the like, but it is extremely difficult to use. On the Windows system, even without any operation, the CPU usage rate is around 20%, which is worse than Kaspersky was back in the day. It imposes many restrictions on your system and is very unfriendly to software development.
For example, if you casually write a Windows program with Visual Studio, once compiled, it is directly killed, and even some Python scripts, if they contain download functions, are considered viruses, which is the type that would rather kill a thousand innocents than let one go.
In terms of functionality, it is far inferior to MSE. This kind of thing is a disaster when used in software development companies.
Combined with the company's domain management and domain policies, this thing cannot be turned off, and it forces updates, and even forces you to restart for updates.
And I don't know how this company does its public relations, now European and American companies, especially American companies, have it installed as a standard, and this company's market value is also very high (80 billion US dollars). I seriously doubt whether this company's market value matches its technical capabilities, and I even suspect that it has achieved such a large market through commercial bribery.
It goes way beyond the tech lever.
Work / business as a system needs a new paradigm and model.
The paradigm lever is what sets new theories, assumptions, values, and principles for a system.
Right now we are treating tech as if it’s a founding value as opposed to a tool for getting a set of tasks done.
This is what explains that while we keep releasing tech that is supposed to improve, the underlying systems are backsliding.
Tech is the wrong tool / system lever for the job of envisioning a new paradigm and model.
Hence, all of the symptoms and breakage going on.
I think you're quite wrong...the pyramids were designed by geniuses to have extraordinary structural integrity. Are you suggesting that Microsoft Windows will last 2000 years?
You are right that the pyramids (particularly ones like the Great Pyramid at Giza) are a very poor analogy for brittle, illogical, error riddled software.
In fact, one would be hard pressed to pick a worse analogy. I bet even ChatGPT could tell us that much.
And MS Windows lasting 2000 years?
Ha ha ha . It’s very lucky to last 2000 minutes before it crashes.
Although Windows as a burial chamber (The Blue screen of Death) IS in keeping with the pyramid analogy.
My wife, at O'Hare airport, said that the Arrivals / Departures boards were all showing the blue screen with a frowny-face and a message like "Unrecoverable error." I have a sci-fi story somewhere in which the characters travel to a distant planet and find robots that seem eerily familiar...until they take one apart and find MS Windows patches in its OS.
The apt analogy for MS Windows and most other software is a patchwork quilt (certainly not an engineering marvel like a pyramid)
Well, after all, that’s what death effectively is, isn’t it?
An “unrecoverable error”
Eg, Generative Pyramid Transformers
Then again, lots of software IS effectively a “pyramid scheme”
Thanks for this. A fundamental problem is that the software industry has no real incentive to clean up their act. Last year we saw a repeat of the old financial industry two step in which they wring their hands before a Congressional hearing about how they need regulation while simultaneously lobbying for none. And our legislators are nearly clueless when it comes to anything technical. Their approach is to associate expertise with corporate stature, when in fact most of the suits are motivated only by their desire for money and power.