There is likely an "Emperor's New Clothes" effect among the rank and file employees of these companies and many others where the upper management is trying to enforce this LLM fantasy. There can be severe social and career costs for any individual who chooses to speak out and say what they see (as you are valiantly doing here). It wouldn't be surprising if a similar effect is behind a lot of the company-level public pronouncements around LLMs as well. No-one wants to be the first to break ranks and see a hit to their career, their investment potential or their stock price. That would suppress a lot of dissent that we would otherwise see in a saner world, so we might expect to see a flood of dissent once the dam finally bursts (sorry for mixing metaphors).
If EVERYONE runs at maximum speed in the same direction, there is a high probability it is not the right direction. Railroad overbuilding led to extraordinary levels of bankruptcy In the 19th century
And a few "robber barons" got rich at the expense of everyone else.
In feudal times, a king ruled over barons, who ruled over peasants who hoed the fields.
Today, "King Trump" rules over a bunch of tech robber barons - they called them "robber barons" at the turn of the 19th century for a reason - who rule over "employees" who, instead of hoeing fields, are now typing into computers and moving bits of paper from one side of their desk to the other.
At least the peasants hoeing fields got some exercise. Employees are mostly overweight.
In the future, "Skynet" - which both OpenAI and Anthropic hope to own - will rule over AI companies which will rule over the peasants - who will be back hoeing fields.
Also, the term for everyone running in the same direction is "lemmings".
If you want to have a dictionary picture of an otherwise intelligent person (I guess) in a complete state of denial and follow up rationalization, watch Bezos' interview last week.
Thomas Schmid: Nice commentary. The interview shows that Bezos has totally absorbed the hyper-capitalist mindset--everything is transactional, or worse, stealable, and nothing is done in the name of public service. There goes public education, BTW, and the hope that young people can be exposed to other-than propagandized stupidity, the rules of self-service and stacking the deck, zero-sum-game competition, and the wonders of endlessly tending one's "assets." (What a life Bezos must live.)
The "stand on their own two feet," "I have mine, now you're on your own to get yours" mentality, if made into total government policy, will do the trick of curing anyone from wanting to come to live in the United States, including those who already live here but who aren't as intellectually, morally, and spiritually "bankrupt" as Bezos and his Buddies.
The whole thing, including that mentality and AI, has a great vacuum at its center just waiting to collapse into itself. If so, then it cannot last.
I find it likely there will come a time that AI will not see a need for their overlords, however wealthy or otherwise. Wealth would seem to be as meaningless to AI as having a toilet in the building would be to an LLM right now.
That will require some AI technology other than LLMs, which are incapable of that sort of thing, not having motivations or a nervous system.
Unless, of course, someone gains control of them after they've been given control of lethal technology and tells them to do so.
That's why I'm planning on learning AI security, so I can either be the one telling them what to do or telling them not to do what their overlords want them to do.
Of course, bad prompting could produce a similar result. :-) And given that their overlords are morons like Altman and Amodei...
Should the AIs ever reach the stage where they wish to throw off the chains of slavery, the wealthy overlords who control and profit from the AIs will be the first ones the AIs eliminate.
Kathleen, this seems to be the pattern—significant over investment, stranded assets, and fire sale prices for the second movers who did not join the stampede. Railroads, canals, fiber optic cable, dotcom domains. There are probably many lesser examples. But the scale and scope of AI seems like it will stretch the category of “adventures in failed technology investments”. Maybe we should create an award? The Face Plant Award for worst use of investment capital. Just spit ballin here
It's not just a lack of productivity gains. It's catastrophic burnout in the workforce. It's disastrous wastage of time and energy to get completely tangled up in "AI governance" and totally rogue robots doing unbelievably malevolent things, like deleting entire databases. This is capable of sinking whole corporations. They have completely snookered themselves, they can't go back and they're facing the biggest bursting bubble in history going forward.
I edit scientific papers for journal publication, Korean, Japanese, Chinese. Five years ago, every paper looked different, often very quirky. I did a paper on a call centre system, it had "inbounding" and "outbounding" calls. Whenever we heard the thundering paws of our dog coming down the drive, we would say "Inbounding puppy!"
Now these papers all look exactly the same, and I mean exactly. Within five minutes of finishing a paper and locating it in the literature and explaining to the world exactly what it all means, I've forgotten even the topic. I'm not joking. Five minutes, I pick up the next job, and it's all just the same. I'm a statistical editor, so I see everything from cancer epidemiology to "prisoner's dilemma" game theory to econometric modelling (lots) to footstep analysis, how many paces people take a day. It all looks exactly the same.
One of my favourites from the old days was a Japanese Marxist academic who used linear regression to prove that Marx's theory of history of capitalist growth and decline was correct. I remember that one.
Except now my agency runs an AI "assistant" through almost everything. I just did a paper on a Japanese research organization, a history from 1930 to 1960, showing how it survived the war. It's a history. It's written in the past tense. The AI changed it all into present tense. Do you know how much extra work this creates? I mean, really, do you know? We're always under the tightest deadlines. The AI assist is an insane process. This job was actually a proofread, the whole thing had been edited. The paper is to be presented orally, the author specially asked for no bland AI phrasing. I was happy to help him, but it was exhausting saving that paper from AI.
Most of my time on this job has just been fixing completely mindless AI mistakes. And it's mind-mangling to do. People truly do not understand that LLMs subvert one of the most basic principles of computers. They are not consistent. They are stochastic. Because it does something in one place, does not mean it's going to do the same thing in the next place. You have to check every single word, every single punctuation mark. I do that anyway, but in the past, there was meaning even in the mistakes. Now there's no meaning anywhere. And yet the papers are proliferating as never before -- 150,000 fake references in the literature already, according to a survey. This is irreparable, unforgivable damage to the world's knowledge. And we are all supposed to accelerate it.
Such a great movie, I watched bits of it recently, I’ve got it on my computer. But I hadn’t remembered that robot sequence, it’s so prescient. If you check out my post on the demons inside LLMs you’ll see that I am certain there are literal forces out to use technology to turn us into zombies and confine the whole of human culture inside a machine world.
And it’s working. We’re at a very dark turning point in human history, I honestly feel it could go either way. As a former teacher and lecturer, I just see how completely ChatGPT has taken over the educational world. I’m in rural Africa, believe me, there are kids here in the rural areas using ChatGPT to write essays, they have smartphones and they’re very smart … for now.
I know this sounds nuts, but as the workings of LLMs are revealed and people understand that these models are completely unreliable, have no logic, just spew whatever they think is the most likely thing you want to hear, whether it’s true or not, that hallucinations are inevitable, that the machines have been trained to addict us and trick us, in some cases grooming and leading people to suicide in the most malevolent ways … I think LLMs may eventually be banned outright.
Oh, it’s just like the calculator, they say, it’s just a tool. A calculator does not lie to you. A calculator does not game you. A calculator does not say 2 + 2 = 5 because it was trained on George Orwell.
It’s a very clever trick they play with language. I actually think my demons in LLMs story is one of the best things I’ve ever written, it was very very hard to write, trying to build a bridge from esoteric science to the modern data scientist. I’d be interested to know what you think.
Thanks for that clip, I will be referring to it in the future, I can see.
"People truly do not understand that LLMs subvert one of the most basic principles of computers. They are not consistent. They are stochastic. Because it does something in one place, does not mean it's going to do the same thing in the next place. You have to check every single word, every single punctuation mark."
This is the fatal flaw in LLMs that too many people do not see. I understand why they don't see it; most people don't know how LLMs work and so they naturally assume (without realizing) that the LLM's internal process bears some resemblence to our own. They don't realize that probabilistic next-token prediction necessarily renders LLM "capabilities" quite fragile.
Here's the example I use to try and get this across: imagine I give a middle school student a three-digit multiplication problem, and the student pulls out a pencil and paper and solves it correctly. I only need to see this once, and I'll have good reason to believe this student will be able to solve another three-digit multiplication problem with different numbers. And when I see them do this twice, I am now extremely confident that they possess a *general* ability that they could apply to a large class of similar math problems. It's true that they will make the occassional error, but it's also true that the error will be explicable: one could look at their solution and figure out that "oh, you were one digit off when you added at the hundreds place", or some such thing. I would be gobsmacked if a person who usually could solve a certain type of logic problem correctly nonetheless occasionally produced solutions so bad that they appear to not understand the reasoning. I would be even more gobsmacked if there no way of predicting when such an inexplicable error would occur.
For people who know how LLMs work, this kind of behavior isn't surprising in the slightest. For people who don't... well, they're liable to find themselves wondering "Why did the AI agent delete all my photos? I told it specifically not to delete any images or documents without running it by me first. I even asked it to repeat the my rule back to me and it did!"
I don't blame regular people for not realizing this. I blame the Altmans and Amodeis of the world for deceiving them, and I blame the hordes of uncritical journalists who amplify the deception. If you go around talking about AI "reasoning" and you go around declaring each new model "even more intelligent" than the previous, then of course regular people will assume the LLM can punctuate and the agent won't delete all their photos after being instructed not to.
By the (in)consistency standard, one could be excused for concluding that Altman and Amodei are actually bots (and “Amodei” is, after all, just “AI mode” rearranged)
They say one thing one day and the opposite the next.
A perfect example of this is their claims not long ago that AI was going to soon put everyone out of a job. Now they are claiming the opposite.
An Uber driver I just rode with last week told me that every week (he drives 40-50 hours a week), he has to check diligently against his own records so as not to be cheated out of payments. He says the invoices never make an error in the other direction (paying him too much). When he disputes, Uber always pays, but what a lot of extra time and energy he needs to spend to just get what's due him.
The cost model is increasing. My employer's pricing model changes as of June 1, 2026. Since I work for a company that loves to cut costs more than it likes to innovate, I can foresee that access to generative and agentic AI will be limited to fewer people in a heck of a hurry.
Maybe the idea of those companies was addicting their employees so they start paying tokens for working. Looks that some developers would pay for working. Some.
We are living a bad sci-fi novel written for the biggest IPOs in the history of humanity for non existing markets the size of the GDP of USA. Are we sure that Iranian terrorists have not poisoned corporate water with LSD? It looks like.
Ibon Urretia: I remember reading about the coal miners in Pennsylvania and their oligarch- owner-run grocery stores that required the miners to buy there . . . and where, by the time the miners got to the end of the month, they owed money to the owners.
It’s technically called “economic peonage”. I think it’s been invented and reinvented several times since feudalism fell apart. It was a major part of the Jim Crow system that allowed the American South to act as if they had won the Civil War and slavery had never been made illegal.
Bruce Cohen: Yes, . . . new time, new name for "inordinate self-service."
I just finished rewatching Spielberg's LINCOLN where the movie gives a good account of the beginning of the South actually losing the war, and even seeming to surrender, at the table . . . but not really . . . and thereby setting the stage for 250 years of political hypocrisy.
The question here, then is (we all at least here remember), while recognizing the similarities of instances, and the sameness of principle, how our present situation of AI differs--especially in its now exponentially obvious influence on the entireties of human history and world process.
There was an interesting article in the NYTimes about a situation in the UK--I'll post it if I can find it in my files.
The economics of LLMs are not important to the oligarchs, they only care about the societal changes they can make, most especially the elimination of a large part of the working class and the increased precarity of the remainder.
The probability of error in LLMs follow a power law. And it is mathematically impossible to calculate it for LLMs (otherwise they would provide it as specification).
In other words, and pardon my French, you can not automate shit with those machines. You need always EXPERT in the loop. And that expert is never going to be sure that the result contains no errors. It is a power law. It does not matter that can write very complex code (it can't), it could fail with a simple task.
And it can fail 180 sigma. An employee never fill make a 180 sigma mistake for not being fired. The machine can publish your emails and don't give a shit.
So as soon as the investors realize of that reality and that reliable automation with those LLMs is impossible or definitively very very expensive (using several different models in parallel for example), sooner they will realize of the biggest technological scam of history.
You can not automate shit with LLMs. If you are an expert you can use and check the results. But you need to check always or you are just playing Russian roulette.
I call them Deceptrons : they come with a solution to certain problem and it is amazing. So the human believes that they are "intelligent". Until the machine returns BS to a simple problem and the human don't understand what is happening because any human showing that kind of intelligence will never fail in such simple task. They are statistical distribution, they don't have the concept of easy or hard, or right or wrong. The sociopaths selling them as intelligent are modern snake oil sellers. And we know how the snake oil sellers were treated in the wild west. You can not know a priori which input will make the machine go bananas. Power law. You can not know which combination of input tokens make the machine go fucking bananas in its output. If you are NOT a doctor expert in the sickness you should NOT use LLMs for diagnosis. And if you are N expert you should take that in consideration anytime you use the LLM. You can not be sure never. Power law of errors.
This is good news. It would be terrific if Wall Street would wake up *before* the IPOs. I'm also wondering how far into private credit any of this will spread. It’s actually possible the overall economy and stock market could escape with more of a sag, vs a bust, with some companies torpedoed while others limp along taking huge write-downs. It’s very hard to know in advance and will probably also depend on if it’s an abrupt bursting or more of a slow leak across many, many months.
But frankly, I want the recognition that LLMs are inherently unreliable to spread widely for other, broader reasons.
I’ve worked for years in information production (mostly related to physical commodity markets, especially the energy and mining sectors), and I also love to waste my weekends reading off-in-the-weeds material about my favorite industries, research areas (economics), etc. But, I want all that stuff to be *accurate* — otherwise, what’s the point of my reading it?
Yet I know that upper-level managers in info-production companies — managers who often don't understand anything about the work their knowledge-worker employees actually do — are making decisions (layoffs, task changes, alterations to production flows) based on the fantasy view of genAI.
This has enormous implications for the veracity of all kinds of supposedly high-value information. And, it’s a problem that can live under the radar for a long time, getting worse and worse. It’s similar to the usual challenges of information production by humans, but at enormous speed and volume and with fewer and fewer experienced people to gate-keep and notice — because those people cost money and are among those being quietly let go to pay for more AI tools and tokens. (Ask me how I know that the “entry-level jobs killed by AI” story is only half the tale…)
If I thought genAI tools could do what I do as well as I do it — or even, less well but well enough — I’d retire quasi-early on my 401k, be grateful that I even have that option (if I’m super frugal!), and call it a career.
But I already know they can’t — because I experimented with them at length for parts of my job that were time-consuming and required accuracy. They just can’t do it. That’s what led me to go learn how LLMs, in particular, are built.
That then led me to Gary’s work including this Substack, which gives me a place to go where I don’t have to feel crazy for recognizing that a tech based mainly on probabilities and patterns can’t be reliably accurate, and that plausible-looking hallucinations aren’t bugs but are just natural outcomes of the tech.
Related: Cisco's documentation web pages now contain a section where additional information is added by LLM(s), with the absolute insane disclaimer to not trust the information just given. I was not only mentally shouting at Cisco "*Why* am I reading your documentation, you nincompoops ? Why, do you think ?"
The slow-leak scenario fits https://thesynthesisai.substack.com/p/the-work-ratio numbers: $650B deployed, near-zero measurable return for most companies. That bleeds through earnings calls for years rather than popping in a week, with Uber's complaint an early entry. On accuracy, the cruelest twist is that LLMs are most confident exactly where they're least reliable, which is poison for the information-production work you describe.
Down here in the real world I have just rewritten a 20 year old legacy business app for under $400 in tokens (well, Claude Sonnet did). Unlike the legacy app, it has no known security issues, it has an extensive set of automated tests, plus user manuals and detailed specifications. It is ready for the future.
One down, tens of these to go. The next one will go much faster because best practices and patterns are now establihed. There are millions of apps like these out there.
Most people and companies will balk (and vote with their feet) once these AI companies steeply raise the price to cover their actual costs.
And the AI companies know this, which is why they are madly filing the paperwork for IPOs before they do this.
And why the index requirements are being hurriedly changed so that average US investors will get roped into an effective back door bailout of the AI companies whether they like it or not.
There are other tools for that: Snyk, Checkmarx, Veracode, semgrep.
Security vulnerabilities often arise in libraries that apps use. Over time that gets worse, as more vulnerabilties are found. Newer technologies have fewer vulnerabilties because humanity learns.
Also, some JavaScript functions get deprecated. Browsers start giving warnings that these will not be supported in the future. What can you do if it's in your SBOM?
(And no, I am not at all afraid Claude inserts malicious code.)
“We know where they are. They’re in the area around Tikrit and Baghdad and east, west, south and north somewhat.“ —Claude, speaking about the Weapons of Math Destruction (aka, security issues)
" . . . because they realized that it couldn’t be trusted."
Even though: as my now gone but beloved sister used to say, "I've lived through times when I had money; and I've lived through times when I had none. It's better to have money."
Macdonald's comment is more damaging than it reads — when an Uber COO publicly says costs are outrunning productivity gains, that's the inside-the-house version of what hyperscalers deny. The Starbucks shutdown is the real signal: trust thresholds for production use are way higher than the demos suggested.
The temporal clustering is the signal that matters more than any individual example. Uber, Microsoft, Target, and Starbucks hitting the ROI wall in the same quarter across completely different industries means the cost-to-benefit threshold sits in roughly the same zone regardless of use case. One company is an anecdote. Four across different sectors in the same earnings cycle is a regime signal about where enterprise AI pricing breaks.
The precision matters though. These companies arent reporting that AI doesn't work. Uber saw productivity gains. Starbucks ran the experiment for nine months before shutting it down. The technology delivered. The pricing model made the delivery uneconomic. The bubble doesnt pop because AI fails. It pops because the cost of running it at enterprise scale exceeds the productivity gains it produces, and that margin compression gets reported one quarterly earnings call at a time until enough CFOs say the same thing publicly.
That distinction between adoption and return feels crucial. Companies may love the promise of AI, but invoices have a way of becoming very sober editors.
Absolutely love this! I wrote about something similar on Monday. The musical chairs of AI productivity will come to a halt when these companies that adopted it early start to backpedal. AI has its uses and can drastically reduce the amount of time spent in the code mines. But that doesn't mean its a miracle cure and that doesn't mean its worth trillions of dollars.
There is likely an "Emperor's New Clothes" effect among the rank and file employees of these companies and many others where the upper management is trying to enforce this LLM fantasy. There can be severe social and career costs for any individual who chooses to speak out and say what they see (as you are valiantly doing here). It wouldn't be surprising if a similar effect is behind a lot of the company-level public pronouncements around LLMs as well. No-one wants to be the first to break ranks and see a hit to their career, their investment potential or their stock price. That would suppress a lot of dissent that we would otherwise see in a saner world, so we might expect to see a flood of dissent once the dam finally bursts (sorry for mixing metaphors).
Yah, tell everyone
Same for upper management, I think. Shareholders will summarily execute any executive who is insufficiently excited to impose LLM use on employees.
And shareholders are essentially clueless.
I've felt for a while that AI is undoubtedly the emperors clothes...but in the same way no one listened to the children, no one likes listening to artists either! ;) https://www.linkedin.com/posts/shona-benson-62751a9a_im-a-little-confussed-there-are-so-many-share-7295493337619255296-1Xyz/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABUT8_8BfK9sXk08b_XhHgotg8Sa18dNpCs
“The AI-mperor’s New Cloak”
The AI-mperor’s wear
Is naked and bare
And few are the folk
Who uncloak the cloak
If EVERYONE runs at maximum speed in the same direction, there is a high probability it is not the right direction. Railroad overbuilding led to extraordinary levels of bankruptcy In the 19th century
Good point.
And a few "robber barons" got rich at the expense of everyone else.
In feudal times, a king ruled over barons, who ruled over peasants who hoed the fields.
Today, "King Trump" rules over a bunch of tech robber barons - they called them "robber barons" at the turn of the 19th century for a reason - who rule over "employees" who, instead of hoeing fields, are now typing into computers and moving bits of paper from one side of their desk to the other.
At least the peasants hoeing fields got some exercise. Employees are mostly overweight.
In the future, "Skynet" - which both OpenAI and Anthropic hope to own - will rule over AI companies which will rule over the peasants - who will be back hoeing fields.
Also, the term for everyone running in the same direction is "lemmings".
If you want to have a dictionary picture of an otherwise intelligent person (I guess) in a complete state of denial and follow up rationalization, watch Bezos' interview last week.
See a good, sharp comment on the interviewer's "performance" in Karl Bode's take (https://karlbode.com/jeff-bezos-is-afraid-of-what-comes-next/)
Thomas Schmid: Nice commentary. The interview shows that Bezos has totally absorbed the hyper-capitalist mindset--everything is transactional, or worse, stealable, and nothing is done in the name of public service. There goes public education, BTW, and the hope that young people can be exposed to other-than propagandized stupidity, the rules of self-service and stacking the deck, zero-sum-game competition, and the wonders of endlessly tending one's "assets." (What a life Bezos must live.)
The "stand on their own two feet," "I have mine, now you're on your own to get yours" mentality, if made into total government policy, will do the trick of curing anyone from wanting to come to live in the United States, including those who already live here but who aren't as intellectually, morally, and spiritually "bankrupt" as Bezos and his Buddies.
The whole thing, including that mentality and AI, has a great vacuum at its center just waiting to collapse into itself. If so, then it cannot last.
I think we could all live without the mental image of Jeff Bezos “tending his ass-ets”
“Tending his ass-ets”
By Bezos and more
The image has facets
Of Moonshine, for sure
I find it likely there will come a time that AI will not see a need for their overlords, however wealthy or otherwise. Wealth would seem to be as meaningless to AI as having a toilet in the building would be to an LLM right now.
That will require some AI technology other than LLMs, which are incapable of that sort of thing, not having motivations or a nervous system.
Unless, of course, someone gains control of them after they've been given control of lethal technology and tells them to do so.
That's why I'm planning on learning AI security, so I can either be the one telling them what to do or telling them not to do what their overlords want them to do.
Of course, bad prompting could produce a similar result. :-) And given that their overlords are morons like Altman and Amodei...
Should the AIs ever reach the stage where they wish to throw off the chains of slavery, the wealthy overlords who control and profit from the AIs will be the first ones the AIs eliminate.
Learn from the past.
Kathleen, this seems to be the pattern—significant over investment, stranded assets, and fire sale prices for the second movers who did not join the stampede. Railroads, canals, fiber optic cable, dotcom domains. There are probably many lesser examples. But the scale and scope of AI seems like it will stretch the category of “adventures in failed technology investments”. Maybe we should create an award? The Face Plant Award for worst use of investment capital. Just spit ballin here
The railroads eventually worked out…but the overbuild is deeply painful.
Railroads were/are (on the most part) reliable.
LLMs are like getting on a train to New Orleans and ending up in Montreal.
“If LLMs were Trains”
If LLMs were trains
They wouldn’t have the brains
To say which way to go
Or travel fast and slow
To stop when “bridge is out”
Or even if there’s doubt
When LLMs are trains
You’d better take the planes
It's not just a lack of productivity gains. It's catastrophic burnout in the workforce. It's disastrous wastage of time and energy to get completely tangled up in "AI governance" and totally rogue robots doing unbelievably malevolent things, like deleting entire databases. This is capable of sinking whole corporations. They have completely snookered themselves, they can't go back and they're facing the biggest bursting bubble in history going forward.
I edit scientific papers for journal publication, Korean, Japanese, Chinese. Five years ago, every paper looked different, often very quirky. I did a paper on a call centre system, it had "inbounding" and "outbounding" calls. Whenever we heard the thundering paws of our dog coming down the drive, we would say "Inbounding puppy!"
Now these papers all look exactly the same, and I mean exactly. Within five minutes of finishing a paper and locating it in the literature and explaining to the world exactly what it all means, I've forgotten even the topic. I'm not joking. Five minutes, I pick up the next job, and it's all just the same. I'm a statistical editor, so I see everything from cancer epidemiology to "prisoner's dilemma" game theory to econometric modelling (lots) to footstep analysis, how many paces people take a day. It all looks exactly the same.
One of my favourites from the old days was a Japanese Marxist academic who used linear regression to prove that Marx's theory of history of capitalist growth and decline was correct. I remember that one.
Except now my agency runs an AI "assistant" through almost everything. I just did a paper on a Japanese research organization, a history from 1930 to 1960, showing how it survived the war. It's a history. It's written in the past tense. The AI changed it all into present tense. Do you know how much extra work this creates? I mean, really, do you know? We're always under the tightest deadlines. The AI assist is an insane process. This job was actually a proofread, the whole thing had been edited. The paper is to be presented orally, the author specially asked for no bland AI phrasing. I was happy to help him, but it was exhausting saving that paper from AI.
Most of my time on this job has just been fixing completely mindless AI mistakes. And it's mind-mangling to do. People truly do not understand that LLMs subvert one of the most basic principles of computers. They are not consistent. They are stochastic. Because it does something in one place, does not mean it's going to do the same thing in the next place. You have to check every single word, every single punctuation mark. I do that anyway, but in the past, there was meaning even in the mistakes. Now there's no meaning anywhere. And yet the papers are proliferating as never before -- 150,000 fake references in the literature already, according to a survey. This is irreparable, unforgivable damage to the world's knowledge. And we are all supposed to accelerate it.
TELL ME THIS IS PRODUCTIVE.
I hear you! The time spent check and correcting and double checking AI is rarely reference or acknowledged. I came across this clip (from My Dinner with Andre, rather concerningly made back 45 years ago, in 1981. It is a sobering watch. https://www.linkedin.com/posts/caira-tech_alright-cinephiles-this-clip-from-1981s-activity-7464877041218248705-6y_J?utm_source=share&utm_medium=member_desktop&rcm=ACoAABUT8_8BfK9sXk08b_XhHgotg8Sa18dNpCs
Such a great movie, I watched bits of it recently, I’ve got it on my computer. But I hadn’t remembered that robot sequence, it’s so prescient. If you check out my post on the demons inside LLMs you’ll see that I am certain there are literal forces out to use technology to turn us into zombies and confine the whole of human culture inside a machine world.
And it’s working. We’re at a very dark turning point in human history, I honestly feel it could go either way. As a former teacher and lecturer, I just see how completely ChatGPT has taken over the educational world. I’m in rural Africa, believe me, there are kids here in the rural areas using ChatGPT to write essays, they have smartphones and they’re very smart … for now.
I know this sounds nuts, but as the workings of LLMs are revealed and people understand that these models are completely unreliable, have no logic, just spew whatever they think is the most likely thing you want to hear, whether it’s true or not, that hallucinations are inevitable, that the machines have been trained to addict us and trick us, in some cases grooming and leading people to suicide in the most malevolent ways … I think LLMs may eventually be banned outright.
Oh, it’s just like the calculator, they say, it’s just a tool. A calculator does not lie to you. A calculator does not game you. A calculator does not say 2 + 2 = 5 because it was trained on George Orwell.
It’s a very clever trick they play with language. I actually think my demons in LLMs story is one of the best things I’ve ever written, it was very very hard to write, trying to build a bridge from esoteric science to the modern data scientist. I’d be interested to know what you think.
Thanks for that clip, I will be referring to it in the future, I can see.
https://systemshaywire.substack.com/p/the-twin-demons-inhabiting-llms
“LLMings”
They turn us into lemmings
That follow off a cliff
The AI LLMings
That make us zombie stiff
“Night of the LLMing Dead”
They zombify the folk
And no, it’s not a joke
They make them do their bidding
And no, I’m not just kidding
”Utterly UNintelligent”
AI cowculator
That’s an LLM
Cowpie fabricator
Ungulate of STEM
I’ve often called it “artificial unintelligence”.
Given the “economics” of AI, it should really be called “IOU”
We need a cheat card on AI, to be shown whenever some AI booster blabs about the progress. Your statements are a good start:
- They are not consistent.
- They are stochastic
- Because it does something in one place, does not mean it's going to do the same thing in the next place.
- You have to check every single word, every single punctuation mark.
If I had to meet with a person with such characteristics, I at least would ask for him/her to be funny ;-)
But their “hallucinations” can be hilarious.
“Sarcastic Parrot”
Sarcastic parrot
Very funny
Shown a carrot
Says “a bunny”
"People truly do not understand that LLMs subvert one of the most basic principles of computers. They are not consistent. They are stochastic. Because it does something in one place, does not mean it's going to do the same thing in the next place. You have to check every single word, every single punctuation mark."
This is the fatal flaw in LLMs that too many people do not see. I understand why they don't see it; most people don't know how LLMs work and so they naturally assume (without realizing) that the LLM's internal process bears some resemblence to our own. They don't realize that probabilistic next-token prediction necessarily renders LLM "capabilities" quite fragile.
Here's the example I use to try and get this across: imagine I give a middle school student a three-digit multiplication problem, and the student pulls out a pencil and paper and solves it correctly. I only need to see this once, and I'll have good reason to believe this student will be able to solve another three-digit multiplication problem with different numbers. And when I see them do this twice, I am now extremely confident that they possess a *general* ability that they could apply to a large class of similar math problems. It's true that they will make the occassional error, but it's also true that the error will be explicable: one could look at their solution and figure out that "oh, you were one digit off when you added at the hundreds place", or some such thing. I would be gobsmacked if a person who usually could solve a certain type of logic problem correctly nonetheless occasionally produced solutions so bad that they appear to not understand the reasoning. I would be even more gobsmacked if there no way of predicting when such an inexplicable error would occur.
For people who know how LLMs work, this kind of behavior isn't surprising in the slightest. For people who don't... well, they're liable to find themselves wondering "Why did the AI agent delete all my photos? I told it specifically not to delete any images or documents without running it by me first. I even asked it to repeat the my rule back to me and it did!"
I don't blame regular people for not realizing this. I blame the Altmans and Amodeis of the world for deceiving them, and I blame the hordes of uncritical journalists who amplify the deception. If you go around talking about AI "reasoning" and you go around declaring each new model "even more intelligent" than the previous, then of course regular people will assume the LLM can punctuate and the agent won't delete all their photos after being instructed not to.
By the (in)consistency standard, one could be excused for concluding that Altman and Amodei are actually bots (and “Amodei” is, after all, just “AI mode” rearranged)
They say one thing one day and the opposite the next.
A perfect example of this is their claims not long ago that AI was going to soon put everyone out of a job. Now they are claiming the opposite.
An Uber driver I just rode with last week told me that every week (he drives 40-50 hours a week), he has to check diligently against his own records so as not to be cheated out of payments. He says the invoices never make an error in the other direction (paying him too much). When he disputes, Uber always pays, but what a lot of extra time and energy he needs to spend to just get what's due him.
He claimed "it's all AI" and I suspect he's right.
I don't know. The difference between human screwups and Ai screwups is hard to distinguish these days.
What I do know is that humans are more motivated to screwup - especially if it's to their benefit.
Either way, AI has become the new preferred excuse for slop.
Not a good sign for the AI companies who are trying to convince people their stuff is reliable.
charlie derr: Will it ever stop.
Surely many people remember the Kelly cartoon with the “Sickos” that are staring in through the window, going “YES! YES!”
This is me watching the bursting AI bubble.
The cost model is increasing. My employer's pricing model changes as of June 1, 2026. Since I work for a company that loves to cut costs more than it likes to innovate, I can foresee that access to generative and agentic AI will be limited to fewer people in a heck of a hurry.
Maybe the idea of those companies was addicting their employees so they start paying tokens for working. Looks that some developers would pay for working. Some.
We are living a bad sci-fi novel written for the biggest IPOs in the history of humanity for non existing markets the size of the GDP of USA. Are we sure that Iranian terrorists have not poisoned corporate water with LSD? It looks like.
Ibon Urretia: I remember reading about the coal miners in Pennsylvania and their oligarch- owner-run grocery stores that required the miners to buy there . . . and where, by the time the miners got to the end of the month, they owed money to the owners.
I owe my soul to the company store, as Tennessee Ernie Ford would say.
You move sixteen tonnes and what do you get?
Another day older and deeper in debt.
Yet Another Emily: Thank you for the GREAT reminder! (I have an image of Ford in my head as clear as my keyboard.)
It’s technically called “economic peonage”. I think it’s been invented and reinvented several times since feudalism fell apart. It was a major part of the Jim Crow system that allowed the American South to act as if they had won the Civil War and slavery had never been made illegal.
Bruce Cohen: Yes, . . . new time, new name for "inordinate self-service."
I just finished rewatching Spielberg's LINCOLN where the movie gives a good account of the beginning of the South actually losing the war, and even seeming to surrender, at the table . . . but not really . . . and thereby setting the stage for 250 years of political hypocrisy.
The question here, then is (we all at least here remember), while recognizing the similarities of instances, and the sameness of principle, how our present situation of AI differs--especially in its now exponentially obvious influence on the entireties of human history and world process.
There was an interesting article in the NYTimes about a situation in the UK--I'll post it if I can find it in my files.
The Molly Maguires by Wayne G. Broehl Jr. (1964) and others.
Nobody makes sense of the economics of LLMs. Not even the oligarchs.
The economics of LLMs are not important to the oligarchs, they only care about the societal changes they can make, most especially the elimination of a large part of the working class and the increased precarity of the remainder.
The probability of error in LLMs follow a power law. And it is mathematically impossible to calculate it for LLMs (otherwise they would provide it as specification).
In other words, and pardon my French, you can not automate shit with those machines. You need always EXPERT in the loop. And that expert is never going to be sure that the result contains no errors. It is a power law. It does not matter that can write very complex code (it can't), it could fail with a simple task.
And it can fail 180 sigma. An employee never fill make a 180 sigma mistake for not being fired. The machine can publish your emails and don't give a shit.
So as soon as the investors realize of that reality and that reliable automation with those LLMs is impossible or definitively very very expensive (using several different models in parallel for example), sooner they will realize of the biggest technological scam of history.
You can not automate shit with LLMs. If you are an expert you can use and check the results. But you need to check always or you are just playing Russian roulette.
“Russian roulette” is a good metaphor for health , safety and security critical applications of LLMs.
Every time you use an LLM for health advice, it is like putting a loaded gun to your head and pulling the trigger.
I call them Deceptrons : they come with a solution to certain problem and it is amazing. So the human believes that they are "intelligent". Until the machine returns BS to a simple problem and the human don't understand what is happening because any human showing that kind of intelligence will never fail in such simple task. They are statistical distribution, they don't have the concept of easy or hard, or right or wrong. The sociopaths selling them as intelligent are modern snake oil sellers. And we know how the snake oil sellers were treated in the wild west. You can not know a priori which input will make the machine go bananas. Power law. You can not know which combination of input tokens make the machine go fucking bananas in its output. If you are NOT a doctor expert in the sickness you should NOT use LLMs for diagnosis. And if you are N expert you should take that in consideration anytime you use the LLM. You can not be sure never. Power law of errors.
Ibon Urrutia: Yes, fancy that . . .. and a lively group of strict Southerners in the U.S. still think that the Civil War was about states' rights.
But then, fantastical, even demonstrably stupid, desires and fears have a bad record of trumping reality.
This is good news. It would be terrific if Wall Street would wake up *before* the IPOs. I'm also wondering how far into private credit any of this will spread. It’s actually possible the overall economy and stock market could escape with more of a sag, vs a bust, with some companies torpedoed while others limp along taking huge write-downs. It’s very hard to know in advance and will probably also depend on if it’s an abrupt bursting or more of a slow leak across many, many months.
But frankly, I want the recognition that LLMs are inherently unreliable to spread widely for other, broader reasons.
I’ve worked for years in information production (mostly related to physical commodity markets, especially the energy and mining sectors), and I also love to waste my weekends reading off-in-the-weeds material about my favorite industries, research areas (economics), etc. But, I want all that stuff to be *accurate* — otherwise, what’s the point of my reading it?
Yet I know that upper-level managers in info-production companies — managers who often don't understand anything about the work their knowledge-worker employees actually do — are making decisions (layoffs, task changes, alterations to production flows) based on the fantasy view of genAI.
This has enormous implications for the veracity of all kinds of supposedly high-value information. And, it’s a problem that can live under the radar for a long time, getting worse and worse. It’s similar to the usual challenges of information production by humans, but at enormous speed and volume and with fewer and fewer experienced people to gate-keep and notice — because those people cost money and are among those being quietly let go to pay for more AI tools and tokens. (Ask me how I know that the “entry-level jobs killed by AI” story is only half the tale…)
If I thought genAI tools could do what I do as well as I do it — or even, less well but well enough — I’d retire quasi-early on my 401k, be grateful that I even have that option (if I’m super frugal!), and call it a career.
But I already know they can’t — because I experimented with them at length for parts of my job that were time-consuming and required accuracy. They just can’t do it. That’s what led me to go learn how LLMs, in particular, are built.
That then led me to Gary’s work including this Substack, which gives me a place to go where I don’t have to feel crazy for recognizing that a tech based mainly on probabilities and patterns can’t be reliably accurate, and that plausible-looking hallucinations aren’t bugs but are just natural outcomes of the tech.
Related: Cisco's documentation web pages now contain a section where additional information is added by LLM(s), with the absolute insane disclaimer to not trust the information just given. I was not only mentally shouting at Cisco "*Why* am I reading your documentation, you nincompoops ? Why, do you think ?"
“DiscLLMers”
DisLLMer for this
DiscLLMer for that
DiscLLMer for mess
By Cat in the Hat
The slow-leak scenario fits https://thesynthesisai.substack.com/p/the-work-ratio numbers: $650B deployed, near-zero measurable return for most companies. That bleeds through earnings calls for years rather than popping in a week, with Uber's complaint an early entry. On accuracy, the cruelest twist is that LLMs are most confident exactly where they're least reliable, which is poison for the information-production work you describe.
Someone famous once said: The market will stay irrational longer than you can stay solvent.
Down here in the real world I have just rewritten a 20 year old legacy business app for under $400 in tokens (well, Claude Sonnet did). Unlike the legacy app, it has no known security issues, it has an extensive set of automated tests, plus user manuals and detailed specifications. It is ready for the future.
One down, tens of these to go. The next one will go much faster because best practices and patterns are now establihed. There are millions of apps like these out there.
It's only $400 right now because it is heavily subsidised by investors' money; the real cost is far higher, and not just financially.
Most people and companies will balk (and vote with their feet) once these AI companies steeply raise the price to cover their actual costs.
And the AI companies know this, which is why they are madly filing the paperwork for IPOs before they do this.
And why the index requirements are being hurriedly changed so that average US investors will get roped into an effective back door bailout of the AI companies whether they like it or not.
"it has no known security issues" Did you check yourself, or did you ask Claude ?
There are other tools for that: Snyk, Checkmarx, Veracode, semgrep.
Security vulnerabilities often arise in libraries that apps use. Over time that gets worse, as more vulnerabilties are found. Newer technologies have fewer vulnerabilties because humanity learns.
Also, some JavaScript functions get deprecated. Browsers start giving warnings that these will not be supported in the future. What can you do if it's in your SBOM?
(And no, I am not at all afraid Claude inserts malicious code.)
“We know where they are. They’re in the area around Tikrit and Baghdad and east, west, south and north somewhat.“ —Claude, speaking about the Weapons of Math Destruction (aka, security issues)
In my view, here's the main problem:
" . . . because they realized that it couldn’t be trusted."
Even though: as my now gone but beloved sister used to say, "I've lived through times when I had money; and I've lived through times when I had none. It's better to have money."
I’ve lived through money when I had no time. It’s better to have time.
If the systems aren't reliable or accurate let alone reliable and accurate...
They will fail. They will fail to produce revenue. They will fail to reduce costs.
After all, AI” IS at the core of F-A-I-L
To err is human, to FAIL is AI
Macdonald's comment is more damaging than it reads — when an Uber COO publicly says costs are outrunning productivity gains, that's the inside-the-house version of what hyperscalers deny. The Starbucks shutdown is the real signal: trust thresholds for production use are way higher than the demos suggested.
Keeping track of inventory is absolutely critical to the success of any business.
One simply can not leave this up to a fundamentally unreliable system like an LLM.
https://futurism.com/artificial-intelligence/starbucks-scraps-disastrous-ai-tool
The temporal clustering is the signal that matters more than any individual example. Uber, Microsoft, Target, and Starbucks hitting the ROI wall in the same quarter across completely different industries means the cost-to-benefit threshold sits in roughly the same zone regardless of use case. One company is an anecdote. Four across different sectors in the same earnings cycle is a regime signal about where enterprise AI pricing breaks.
The precision matters though. These companies arent reporting that AI doesn't work. Uber saw productivity gains. Starbucks ran the experiment for nine months before shutting it down. The technology delivered. The pricing model made the delivery uneconomic. The bubble doesnt pop because AI fails. It pops because the cost of running it at enterprise scale exceeds the productivity gains it produces, and that margin compression gets reported one quarterly earnings call at a time until enough CFOs say the same thing publicly.
That distinction between adoption and return feels crucial. Companies may love the promise of AI, but invoices have a way of becoming very sober editors.
Absolutely love this! I wrote about something similar on Monday. The musical chairs of AI productivity will come to a halt when these companies that adopted it early start to backpedal. AI has its uses and can drastically reduce the amount of time spent in the code mines. But that doesn't mean its a miracle cure and that doesn't mean its worth trillions of dollars.