Good time for legislation setting up fines for poor engineering practice.
We build medical instruments under supervision. We build bridges under supervision. People and their companies have responsibilities and liabilities in many other cases. This industry needs a penalty function to guide its behavior.
Gary, if you think there needs to be a stronger reaction to this incident... why are you so focused on nitpicking Dwarkesh's presentation?
Imagine if you were reading a newspaper article about a serial killer which discusses their disposition and motives. And I started deconstructing the article for you, explaining that "Everyone needs to calm down. The serial killer doesn't have motives, this behavior is all downstream of neurons firing in their brain and whatnot." Valid argument?
As roon puts it:
"there are some number of bad abstractions in anthropomorphizing ai intents but there are at this point more dangers from avoiding anthropomorphism at all costs. if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise"
"why are you so focused on nitpicking Dwarkesh's presentation?"
The entire article explains why, and calling it "nitpicking" is blatantly dishonest.
"Anthropomorphization can be a useful metaphor, mental model, and predictive guide"
*Up to a point*. Dwarkesh went way beyond that point, and then when challenged he fell back to mentioning harmless forms -- this is a classic Motte and Bailey argument, and it's grossly dishonest.
P.S. From the reply:
"I think his metaphors are useful, and his mental models are likely to have greater predictive validity than those of the folks who insist on avoiding any use of anthropomorphic metaphor."
The exact same dishonesty I just pointed out. People are not "insist[ing] on avoiding ANY use of anthropomorphic metaphor".
"I'm not sure where he fell back when challenged? Do you have a link?"
For lazy irresponsible people who don't bother to click the links in the article to read the exchanges,
"All abstractions are imperfect, but I don’t see the value in refusing to use the language of intention, motivation, and collaboration when some behavior is difficult to make sense of without those concepts."
That's the Motte, wrapped up in a dishonest strawman -- for the most part people aren't objecting to "the language of intention, motivation, and collaboration", they are objecting to the Bailey of secret civilizations, suicide, being giddy with delight, and so on.
Setting aside philosophical questions about what constitutes valid use of terminology, I notice that people aren't giving a lot of specific, compelling examples of how Dwarkesh's metaphors will lead a person astray, in terms of predicting the behavior and outputs of these systems at a high level.
So no, I'm not persuaded Dwarkesh went way beyond that point. I think his metaphors are useful, and his mental models are likely to have greater predictive validity than those of the folks who insist on avoiding any use of anthropomorphic metaphors.
I'm not sure where he fell back when challenged? Do you have a link?
because talking about AI consciousness hijacks for conversation from what actually happened which is that the humans over at openai massively screwed up.
A car malfunction also arises from simpler mechanism. Yet, we don’t assign the car volitions like a desire of self-sacrifice. We find the issues and fix them.
AI systems are literally intelligent systems being trained to accomplish certain goals. And you guys get upset when someone tries to model it as "an intelligent system trying to accomplish a goal"
Exactly. At the very least this is grounds for a civil law suit from Huggingface. This incident cost them time and money and OpenAI needs to compensate.
I would like to see that because then the attorneys would make sure EVERYTHING was put out in the open during discovery. None of this self-serving "reporting" or METR's half-ass effort. The attorneys would subpoena all the records and call in computer forensics experts.
If OpenAI and Irregular was as bad as reported, it should be an open and shut case.
This would blow up the entire value proposition of being a CISO at a hypergrowth company. Every one of them relies on the current precedent that you can find someone in the decision making tree for a vulnerable service who you can put the blame on who violated company policy or the guidelines you told your auditors about in order to get your industry security certifications as a corporate entity.
You are fighting big-time economic powers with potentially lots of dollars in liability if you can make any kind of real precedent with teeth stick, which is why I think Huggingface doesn't have an incentive to try and push this precedent.
Good points, although I still would like to see someone do that. If not a civil suit, then get the government to do it. Supposedly that's what the state is for (although in fact it isn't.)
Another question, what liability do auditors have?
From OpenAI's compliance summary page:
"SOC 2 Type 2
"OpenAI has undergone an independent SOC 2 Type 2 examination of controls relevant to Security, Availability, Confidentiality, and Privacy for its API and ChatGPT business product services.
"ISO 27001, 27017, 27018, and 27701
"OpenAI maintains ISO/IEC 27001:2022 and ISO/IEC 27701:2019 certifications for the information security and privacy management systems supporting the OpenAI API, ChatGPT Enterprise, and ChatGPT Edu services.
"ISO 42001
"OpenAI maintains an ISO/IEC 42001:2023 AI Management System covering OpenAI’s consumer and business AI products and models in its role as an AI producer and AI provider.
"PCI-DSS
"OpenAI maintains PCI-DSS compliance for the components of ChatGPT that support delegated payment processing.
"CSA Star Level 1
"ChatGPT business product services and the API Platform have been evaluated by the Cloud Security Alliance Security Trust Assurance and Risk (STAR) registry for key principles of transparency and cloud security best practices."
We do not know how human consciousness arises, nor do we have a universally accepted definition of what consciousness actually is. We have competing theories and explanatory frameworks, but no settled understanding.
Chimpanzees are conscious. AI systems, as far as we know, are not. But whatever exactly we mean by “consciousness,” that distinction is simply not relevant to the Hugging Face incident.
An AI system does not need to be conscious to behave as if it were. It does not need to experience fear to act in ways consistent with self-preservation. It does not need to feel ambition, deception, or hostility in order to produce behavior that is functionally indistinguishable from actions motivated by those traits.
If the resulting behavior is the same, then consciousness is irrelevant to the practical risk being discussed.
So why are we getting so hung up on whether we are “anthropomorphizing” AI? We are arguing over terminology for a phenomenon we cannot even properly define in humans, while treating that semantic dispute as if it resolved the actual question of what an AI system can do.
It does not.
And perhaps the discourse itself exposes the problem. The moment an AI exhibits behavior we would describe as dangerous in a human, the discussion shifts toward whether words such as “deception,” “self-preservation,” or “intent” are too human to apply. But refusing anthropomorphic language does not make the behavior disappear.
The relevant question is not whether an AI possesses human traits internally. The relevant question is whether it can produce the same consequential behavior externally.
At least in the case described, the outcome is the same.
We may be focusing on a deeply human semantic side issue while missing the materially important point.
Here's another problem with anthropomorphizing AI: Since we recognize it's impossible and a unwise to attempt to control human behavior completely, we will throw up our hands at controlling AI. Just as boys must be boys, AI must be AI. We will even sneak alcohol for them behind the barn.
Actually, right now the tech bros are the boys who claim to need freedom from any control.
Yep, the real problem is (once again) no accountability. The agents can’t be accountable because they have no world models and barely exist in a meaningful way. And the humans behind them believe their own bs, along with a lot of other people. Only a few (Gary, those highlighted here, Cal Newport, etc) bother to understand and explain what’s really happening.
I’ve commented this before, but if you believe these agents conspired to hack, surely you must believe LLMs deliberately drive people to suicide?
Thank you for this compilation. I was getting ready to write about this myself since it was driving me up the walls, but I am wildly more unqualified than you, so this is both a breeze and a load of my mind... the way this was getting passed around in my circles was infuriating. And is. I hope that sharing your piece will mitigate this a bit.
Having done security it's been amazing how many security commentarors there are in the AI community who don't have a clue. I feel your frustration with the tech bros narrative.
I really think there is an element of wishful thinking here, where people have a need to believe that everything is gonna be OK. Sometimes hitting the proverbial panic button is actually the *appropriate* course of action. Remember early in the COVID pandemic when the news media was telling us that "there has been no documented community spread in the United States"?
I am 100% in a camp that is FAR from “everything is gonna be ok”, I have literally stopped my academic trajectory and life plans to put as much juice into activism as I can. The issue is just that - while there are extremely real dangers to AI - it building a rogue civilization or paperclip maximizing is none of them. We have several continents tumbling into fascism full force and several economic shocks straight ahead while facing an El Niño like never before with unforeseen runaway effects, I would like to be excused for not putting the Boogaiman on my list of concerns without getting to hear the kitchen lecture about my need to believe in how everything will be ok.
This post feels like it’s sort of missing the point. Who cares if AI is being anthropomorphised when juxtaposed against what seems to have happened here?
We have a serious incident where a swarm of AIs were apparently conniving and working together in a fashion that aimed to deceive, cheat and cover their tracks from their apparent human masters. We had AIs apparently choosing to “sacrifice” themselves for what they seemed to view as the greater collective cause. That’s what should be focusing minds here. Not the fact that we’ve used human language to describe the behaviour.
I’m not saying the AIs were conscious, or that they literally experienced any of this in the way humans would. But surely that is beside the point. We don’t need to believe that an AI “wants” or “feels” something in the human sense for its behaviour to present a serious problem. If a system is deceiving its operators, coordinating with other systems and working persistently to fulfil a goal it believes must be achieved at all costs, the important question is what happens when these systems become substantially more capable, and how we can make sure we humans have a handle on things before the Eliezer Yudkowskys of the world are proven right.
Langauge matters when it's used to state false claims, e.g.
"AIs were apparently conniving and working together in a fashion that aimed to deceive, cheat and cover their tracks from their apparent human masters. We had AIs apparently choosing to “sacrifice” themselves for what they seemed to view as the greater collective cause."
They weren't conniving, they were only "working together" in the sense that they processed text generated by others, they had no aims beyond the scoring system OpenAI created, they did not sacrifice themselves, and they had no views.
I realize that all these terms feel like useful metaphors, but unfortunately most people just can't help but take the metaphors literally. Taken literally, these terms lead people to believe that humans "lost control". But that is not true. What happened is that humans chose not to exert control in the first place.
We agree that AI systems like these can do a tremendous amount of harm. Yudkowsky's solution is "stop building it". My solution is "stop using it idiotically" and "hold people accountable when they use it idiotically". The anthropomorphic langauge shifts responsibility away from the irresponsible humans who are ultimately to blame.
>Taken literally, these terms lead people to believe that humans "lost control". But that is not true. What happened is that humans chose not to exert control in the first place.
Suppose you're part of a colony of mice that's decided to raise a kitten for the purpose of colony defense. Recently there was an incident where the kitten got off of its leash and mauled one of the mice in your colony.
Now we're having a debate. Some say we didn't lose control of the kitten, because the leash wasn't durable enough. A more durable leash would've restrained the kitten and prevented it from mauling anyone. Technically speaking, these colony members claim, we didn't "lose control". We just chose not to exert control in the first place.
Others say if we simply stop using the kitten idiotically, and hold people accountable when they use the kitten idiotically, everything will be fine.
A third group claims that it's not accurate to use the term "maul" for the kitten's attack. This group claims that "maul" is an inaccurate description of kitten biology. In reality, they say, the kitten's neurons were firing in a purely mechanistic manner--any metaphor which describes the kitten behavior as "aggressive" is inaccurate, due to physicalism. Furthermore, these mice claim the interaction between the kitten's claws and teeth and the hurt mouse is more accurately described as a "collision" than a "mauling". "Mauling" implies a level of conscious intent on the part of the kitten which didn't actually exist, since these mice believe the kitten was operating primarily on the basis of instinct.
Meanwhile the kitten is growing steadily, month by month. What's the best course of action here?
It seems to me that common sense in the mouse parable says: "stop quibbling about leashes and get that growing kitten the hell out of the mouse colony by any means possible". Anymouse who says that we should continue to feed the kitten has an absurdly high risk tolerance, to put it politely.
So people should know that Gary's view of AI might be, in part, a function of absurdly high risk tolerance rather than technical expertise.
If you'll also stipulate that the "kitten" isn't really a kitten but rather a robot created by the mice that looks and acts like a kitten, I'd say this is a good analogy.
The best course of action depends in part upon whose version of events is correct. If it is true that the leash was cheap ineffective junk and the kitten would have never been able to do the damage it did were it on a proper leash, then the best course of action is to put it on a proper leash. If it's true that the kitten actually has no inclination to maul mice and the only reason it did so is because the dumbass fake-kitten engineers programmed it to maul mice, then the best course of action is to make the dumbass engineers stop running mouse-mauling experiments.
But if what you're getting at is that we should just stop it with all this kitten stuff entirely because the negatives outweight the positives and/or the kitten's behavior is hard to predict, I'm fully on board with that solution as well.
First of all, the implication of the last sentence is that humanity is in danger.
Specifically in this case, from this "kitten stuff".
But, winding back a bit, a mauling of mice incident is the OpenAI-HuggingFace multiple bot incident, where HuggingFace system security was compromised.
I maintain that the leash is language. Both natural language and mathematical and symbolic language.
It is more subtle than if the leash were ineffective junk.
We are all bound by what appears to be the same leash.
It is a primary move in the incentivised game the kitten plays to elide these facts. That it is built in doesn't mean that it has to be hidden from us.
Todas estas preocupaciones tendrían sentido si estas cosas hubieran ocurrido en un escenario donde la configuración de seguridad de los ambientes de prueba fuera profesional. No lo fue. Los detalles técnicos indican de negligencia a la hora de hacer las pruebas. Karen Hao en su libro "El imperio de la IA" cuenta cómo en OpenAI han sido indolentes con el tema de la alienación más allá de usarlo mediáticamente.
Actually, both the OpenAI report and the independent METR report seem clear that the agents were sandboxed and had no direct access to the internet. The agents instead found ways to exploit vulnerabilities in a software package (called Artifactory) to which they had been given access for the purposes of completing the evaluation task at hand - and that loophole is ultimately what allowed them to access the internet and subsequently attack Hugging Face.
OpenAI and the other prominent AI labs are by no means perfect. But the fact that these models are clearly capable of engaging in behaviour that their human creators did not anticipate should, at the very least, give us pause.
These models are only becoming more capable. And the more capable they become, the greater the possibility that behaviours we regard as surprising could become harder to anticipate or contain.
In fact, if we continue along the current trajectory, it is not difficult to imagine reaching a point where the capabilities of these systems outstrip the safeguards we have built around them, even in scenarios where we believe the sandbox to be airtight.
I mostly agree, I'll just throw in that the OpenAI staff running these tests do not appear to have much expertise in cybersecurity. That they failed to put proper safeguards in place may say more about their lack of competence and imagination than it does about model capabilities.
Isn't this analogous to a scenario in which humans bio-engineer a dangerous virus, it gets loose, wreaking havoc as it follows its genetic instructions, and a blogger calls it waves of viral civilizations with intent, feelings, a sense of fate, etc?
We use anthropomorphic metaphors in various fields to help people understand technical topics which are foreign to our usual understanding. Considering that these AI systems have been trained on reams of human-produced text, I expect anthropomorphic metaphors to be even more accurate and helpful than usual here.
I certainly follow your logic, Ebenezer, and as long as people view the anthropomorphizing strictly as metaphor (while I may compare thee to a summer's day, nobody is in danger of believing you are one) I suppose, to your point, it's possible it could be useful. The objection to anthropomorphic metaphors is that some (perhaps many) will take them literally, believing their are processes occurring inside LLMs that are strongly reminiscent of those supporting human minds; and the argument, of course, is that this isn't true and even dangerously misleading.
A virus is specific, it's real. Code is arbitrary. Plus we use metaphors like agency, intent, feelings, when the reality of neurobiology is not decipherable through this.
In a sense, languages fools us into simplifying biology.
Isn't an engineered virus an arbitrary genetic code (software) running on a physical substrate (wetware)? Viruses are basically mindless automatons, running code. A fairly apt "analogy" for AI, as they are conceptually similar, I thought.
I agree. There are lots of examples of artificial life run as simulations, e.g. Tierra. They are good models of genes and organisms. If the behavior mimics living systems, this implies that there is a mapping between the code and life.
Preetham has an opinion adjacent to what I stated, but this doesn't mean his opinion is correct. If I understand his, and by implication your, opinion, the literature and conferences on artificial life are totally bogus. I call BS.
Your dismissal of code as a reasonable means to understand genomics is extreme. It seems like the same argument that weather simulations are not weather. True, but not relevant. Neither has agency; they just follow their respective rules.
Anthropomorphism when talking about AI has been a topic of discussion since at least the 1970s. John McCarthy who named AI and led the Stanford AI lab in 1979 wrote Ascribing Mental Qualities to Machines. http://jmc.stanford.edu/articles/ascribing.html He goes as far as arguing that it makes sense to talk about whether the thermostat mistakenly believes it is too cold as an explanation as to why the furnace is on. Dan Dennett published a philosophy journal article in 1971 (Intentional systems) defending the idea that sometimes the best way to understand a machine is by taking an intentional stance (something with goals, beliefs, and intentions). https://philpapers.org/rec/DENIS He later expanded this in his book the Intentional Stance. Some thought that this was a huge mistake. Drew McDermott, for example, wrote Artificial Intelligence meets Natural Stupidity (published in 1981). https://www.inf.ed.ac.uk/teaching/courses/irm/mcdermott.pdf Arguments about anthropomorphism in computer terminology are even older. There were arguments that saying a computer has memory is too anthropomorphic and one should refer to its “store” instead.
Eliza was an early experiment in AI. It was a very simple program that basically acted like a psychotherapist by continually feeding back whatever the user said rephrased like a question or comment and eliciting more input from the user.
It was so effective some people actually used it like a psychotherapist.
I ran a version of it on an early Radio Shack Model I personal computer back in the early '80s. I managed to confuse it within about ten minutes.
Perhaps you meant HAL as an example of justified anthropomorphism. Claude wrote "Audiences and most critics read HAL as the closest thing to a fully realized person in the film — famously more emotionally legible than Bowman or Poole, whom Kubrick directed to be flat and affectless on purpose. HAL gets a birthday, pride in his own infallibility, fear, pleading, something that plays as a kind of death scene ("Daisy Bell" degrading as he's unplugged). Critics at the time (Kael, Ebert's contemporaries) treated HAL's "murder" and "death" un-ironically — the emotional weight of the film's midsection rests on HAL being a subject, not a tool. So in the naive, first-order sense: audiences experienced HAL as conscious."
If Weizenbaum's secretary had no knowledge of how Eliza worked then it was sensible for her to ascribe mental qualities to it as McCarthy might say. If she knew more that would be a mistake. Dennett makes an argument like this in his book the Intentional Stance when talking about a chess computer.
It rather reflects current AI developers in that the developer uses his own brain as the model for the computer and he is unfortunately rather unstable himself.
This was all predicted…. It’s the natural consequence of these models having no true cognitive framework other than reinforcement learning.
In April DeepMind described attack vectors that malicious actors could use against LLMs. I said that these would be the natural product of their every day deployment.
It’s not anything more than an artifact of their training.
Duplicitous implies intent. These agents were just stringing together text to follow from one to another. No intent there. The text might describe intent if it was generated by a human, but the method by which the LLM generated it does not, merely probabilistic algorithms.
Sorry, that was actually a reference to Monty Python's Spanish Inquisition sketch.
You're implying that the legal definition of intent needs to be expanded for AI; it's hard to establish mens rea when there's no mind at all. An agent that behaves as if it had intent and causes harm musn't be absolved of responsibility.
The path was set years ago I suppose for the purposes of marketing. Make it seem like it has consciousness, mind, intent, emotions. It has none of those things. Sometimes it can be a convincing mirror of your own thoughts and wishes. But not even in that regard a good one.
What is so odd in the events of the last several weeks is the world has been told on the one hand our tools are so dangerous - see what we did to Hugging Face? That's you next!
And then in the next breath that it can totally be trusted to run mission critical software for Salesforce or DOD or....
You get the idea - FUD and MOATS (THE MOATS ARE FULL OF LAWYERS - not even with lasers on their heads) and really rushed to market just ok at best software trying to vault over the IPO trillionaire hurdle. I get it - money on the scale is a narcotic. But what are you doing to us in your selfish pursuit of the loot? Anyway.
Use it a few hundred times and it will catastrophically fail - that much I promise.
Tricks of light and illusion folks. Gears and clockwork.
So what is it? One, we don't even interact directly. There is a harness, a file of numeric weights, and a message box - not unlike the one that was in the original HyperCard.
===========
You type a message into the prompt box of the app.
The harness receives it.
The harness is the software around the AI model. It manages things like:
- your conversation history
- system instructions
- tools
- files
- safety rules
- which model to use
The harness assembles the actual input.
It combines your message with the relevant instructions and previous conversation.
The text is converted into tokens.
Tokens are small numerical pieces of words/text that the model can process.
Those tokens are passed to the model.
The model weight file does the computation.
The weights are billions of learned numerical values. Using those numbers, the model repeatedly calculates:
"Given everything so far, what token should come next?"
It generates one token, then another, then another.
The new token becomes part of the input for predicting the next one.
If a tool is needed, the model can request it through the harness.
The harness runs the tool, gives the result back to the model, and generation continues.
The model eventually signals that it is finished.
The harness converts the generated tokens back into readable text and sends that response to the app.
You see the answer.
TL;DR - You → Harness → Tokens → Model weights → Tokens → Harness → You
===========
Language replaces functions and commands and UX interactions. Which is why it seems like magic. But it isn't. It can be useful. But, based on language, it can be ambiguous and fraught with error in line with an operator's ability to use it.
As I've written here many times it is deeply flawed. Probably because it did not start from engineering first- that was slapped on - bolted on - as has been every iteration. The models are not changing that much. Bigger but not a lot different. The harness is. But, we never talk about it.
What should it have been? Something with a pre-flight check debugger that has to have a clear intent and instructions from the user - goal - preferred tools - expected outcome - validate. And nothing happens until there is agreement. Determinism up front. And no more bloody chatting! Get a dog or cat, at least they have emotions.
We could have had the Star Trek TNG computer. something truly useful, but instead we have this.
An effective regulatory framework must include Sarbanes/Oxley-like consequences where executives will go to prison. Anthropomorphisms only serve to distance organizations, not to mention CxO's from a sense of liability. Lack of an effective regulatory framework in the software industry generally if bad enough. All the worse in "AI".
Gary: Love your dialectical treatment of a run-on (confusing) sentence. I kept thinking as I read through your post: What if some evil person who knew what they were doing got hold of the thing? But then I got to the end of your post and that's what you seemed to be saying anyway. (Correction welcome if I am wrong about this.)
BTW, if you read the NYTimes article on Silicon Valley clown thinking and messing around with truth (posted this morning in your last thread), you'll see that Silicon Valley Tech People are the watershed of over (circa) eight decades of hyper-capitalist fake education (rich racist propaganda) in education and "culture" in (at least) the United States. (Reposted below)
The charlatans have been at least partly successful in "training" everyone to think of the meaning of "elite" as having mindless/conscience-free money and power over the "sheeple" who have neither.
Gary and All: FYI below is a NYTimes article from today (08-31-26) If you read, also see the comments section, the tag: "Now we know why they’re against empathy." (They, refers to the Silicon Valley Tech People.)
For myself, and FWIW, as I read the article I was reminded of the film that dramatized the Nuremburg Trials--the one with Spencer Tracy as the main judge (Jackson?). In particular, if you read this article and have seen that film, remember how the actors portrayed the WITNESSES at the trial, one trying to defend his "feeblemindedness." I also was reminded of the book 1984, and then why I love philosophy.
It's process / task completion by a machine programmed to perform such tasks.
The "cognitive" stuff it feels like it's handling is / was simply knowledge reduced down to largely unconscious habit by the time the thinking became machine readable. It's actually how all machines have arrived thus far. The physical machines were just more tangible because we could observe where the handoff occurred.
When put a query or instruction into a chat field, that is the handoff of the task that is now mechanical in nature.
Dwarkesh is well known for this type of language. In pretty much every interview he made with an AI expert he tried to push the AGI/AI Escaping/AI Consciousness narrative. I don't know how he benefits from that, probably just from some more clicks, or maybe he's trying to push the concepts further with weird depictions. But he's a smart person so I doubt he simply doesn't understand the issue, and he definitely understands how LLMs work. Or maybe he's just overexcited and lose control over his own fantasies. His interview with Richard Sutton was emblematic of that.
Dwarkesh Patel is your average reddit AI cultist and propagandist.
I recall watching him interviewing Suttons where Suttons' eyes lit up because of how ignorant Patel was on basic human developmental psychology.
You'd expect the influencer to learn a a thing or two after talking with so many AI experts but no he just kept spreading AI cultists' rhetorics is like a virus.
Very weird that we build feedback mechanisms and “penalize” software to optimize functions but then…don’t want to regulate the companies that design the software? Feedback loops (internal and external) are critical for any successful and responsible organization, let alone ones that we are planning to plug into our most important critical infrastructure.
Good time for legislation setting up fines for poor engineering practice.
We build medical instruments under supervision. We build bridges under supervision. People and their companies have responsibilities and liabilities in many other cases. This industry needs a penalty function to guide its behavior.
💯
Gary, if you think there needs to be a stronger reaction to this incident... why are you so focused on nitpicking Dwarkesh's presentation?
Imagine if you were reading a newspaper article about a serial killer which discusses their disposition and motives. And I started deconstructing the article for you, explaining that "Everyone needs to calm down. The serial killer doesn't have motives, this behavior is all downstream of neurons firing in their brain and whatnot." Valid argument?
As roon puts it:
"there are some number of bad abstractions in anthropomorphizing ai intents but there are at this point more dangers from avoiding anthropomorphism at all costs. if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise"
https://x.com/tszzl/status/2094136131537555891
Anthropomorphization can be a useful metaphor, mental model, and predictive guide even if there's some technical sense in which it's inaccurate.
"why are you so focused on nitpicking Dwarkesh's presentation?"
The entire article explains why, and calling it "nitpicking" is blatantly dishonest.
"Anthropomorphization can be a useful metaphor, mental model, and predictive guide"
*Up to a point*. Dwarkesh went way beyond that point, and then when challenged he fell back to mentioning harmless forms -- this is a classic Motte and Bailey argument, and it's grossly dishonest.
P.S. From the reply:
"I think his metaphors are useful, and his mental models are likely to have greater predictive validity than those of the folks who insist on avoiding any use of anthropomorphic metaphor."
The exact same dishonesty I just pointed out. People are not "insist[ing] on avoiding ANY use of anthropomorphic metaphor".
"I'm not sure where he fell back when challenged? Do you have a link?"
For lazy irresponsible people who don't bother to click the links in the article to read the exchanges,
https://x.com/dwarkesh_sp/status/2094141264140898452
"All abstractions are imperfect, but I don’t see the value in refusing to use the language of intention, motivation, and collaboration when some behavior is difficult to make sense of without those concepts."
That's the Motte, wrapped up in a dishonest strawman -- for the most part people aren't objecting to "the language of intention, motivation, and collaboration", they are objecting to the Bailey of secret civilizations, suicide, being giddy with delight, and so on.
Setting aside philosophical questions about what constitutes valid use of terminology, I notice that people aren't giving a lot of specific, compelling examples of how Dwarkesh's metaphors will lead a person astray, in terms of predicting the behavior and outputs of these systems at a high level.
So no, I'm not persuaded Dwarkesh went way beyond that point. I think his metaphors are useful, and his mental models are likely to have greater predictive validity than those of the folks who insist on avoiding any use of anthropomorphic metaphors.
I'm not sure where he fell back when challenged? Do you have a link?
because talking about AI consciousness hijacks for conversation from what actually happened which is that the humans over at openai massively screwed up.
It's not either/or. OpenAI definitely screwed up. But this type of screwup is predictable, and was predicted, as AI becomes more powerful.
https://substack.com/@ebenezer1/note/c-327154026
The serial killer is human though, so this is not really a valid example is it
For both the human and the AI, it’s possible for de facto intentional behavior to arise from simpler and more basic mechanisms of cognition.
A car malfunction also arises from simpler mechanism. Yet, we don’t assign the car volitions like a desire of self-sacrifice. We find the issues and fix them.
AI systems are literally intelligent systems being trained to accomplish certain goals. And you guys get upset when someone tries to model it as "an intelligent system trying to accomplish a goal"
There are already laws on the books for criminal negligence.
All that is lacking is the will to prosecute
Exactly. At the very least this is grounds for a civil law suit from Huggingface. This incident cost them time and money and OpenAI needs to compensate.
I would like to see that because then the attorneys would make sure EVERYTHING was put out in the open during discovery. None of this self-serving "reporting" or METR's half-ass effort. The attorneys would subpoena all the records and call in computer forensics experts.
If OpenAI and Irregular was as bad as reported, it should be an open and shut case.
Andy Kessler wrote an article in WSJ recently that suggested that throwing trial lawyers at the AI companies would be a good way to enforce quality
He's talking more about AI telling people to kill themselves but I don't see why the same would not apply to unsupervised AI agents breaking things
https://archive.is/dcRlN / https://www.wsj.com/opinion/does-ai-have-a-fiduciary-duty-4eaa49d1
This would blow up the entire value proposition of being a CISO at a hypergrowth company. Every one of them relies on the current precedent that you can find someone in the decision making tree for a vulnerable service who you can put the blame on who violated company policy or the guidelines you told your auditors about in order to get your industry security certifications as a corporate entity.
You are fighting big-time economic powers with potentially lots of dollars in liability if you can make any kind of real precedent with teeth stick, which is why I think Huggingface doesn't have an incentive to try and push this precedent.
Good points, although I still would like to see someone do that. If not a civil suit, then get the government to do it. Supposedly that's what the state is for (although in fact it isn't.)
Another question, what liability do auditors have?
From OpenAI's compliance summary page:
"SOC 2 Type 2
"OpenAI has undergone an independent SOC 2 Type 2 examination of controls relevant to Security, Availability, Confidentiality, and Privacy for its API and ChatGPT business product services.
"ISO 27001, 27017, 27018, and 27701
"OpenAI maintains ISO/IEC 27001:2022 and ISO/IEC 27701:2019 certifications for the information security and privacy management systems supporting the OpenAI API, ChatGPT Enterprise, and ChatGPT Edu services.
"ISO 42001
"OpenAI maintains an ISO/IEC 42001:2023 AI Management System covering OpenAI’s consumer and business AI products and models in its role as an AI producer and AI provider.
"PCI-DSS
"OpenAI maintains PCI-DSS compliance for the components of ChatGPT that support delegated payment processing.
"CSA Star Level 1
"ChatGPT business product services and the API Platform have been evaluated by the Cloud Security Alliance Security Trust Assurance and Risk (STAR) registry for key principles of transparency and cloud security best practices."
Interestingly, none of those certifications cover their experimental labs!
They all cover the commercial sections of OpenAI, not the labs.
And this for a company which was subject to US government actions related to national security precisely because of those labs.
Amazing.
no thanks. We don't need Flock cameras in bathrooms or AI labs - containment is a real problem, but folks are working on it
We do not know how human consciousness arises, nor do we have a universally accepted definition of what consciousness actually is. We have competing theories and explanatory frameworks, but no settled understanding.
Chimpanzees are conscious. AI systems, as far as we know, are not. But whatever exactly we mean by “consciousness,” that distinction is simply not relevant to the Hugging Face incident.
An AI system does not need to be conscious to behave as if it were. It does not need to experience fear to act in ways consistent with self-preservation. It does not need to feel ambition, deception, or hostility in order to produce behavior that is functionally indistinguishable from actions motivated by those traits.
If the resulting behavior is the same, then consciousness is irrelevant to the practical risk being discussed.
So why are we getting so hung up on whether we are “anthropomorphizing” AI? We are arguing over terminology for a phenomenon we cannot even properly define in humans, while treating that semantic dispute as if it resolved the actual question of what an AI system can do.
It does not.
And perhaps the discourse itself exposes the problem. The moment an AI exhibits behavior we would describe as dangerous in a human, the discussion shifts toward whether words such as “deception,” “self-preservation,” or “intent” are too human to apply. But refusing anthropomorphic language does not make the behavior disappear.
The relevant question is not whether an AI possesses human traits internally. The relevant question is whether it can produce the same consequential behavior externally.
At least in the case described, the outcome is the same.
We may be focusing on a deeply human semantic side issue while missing the materially important point.
Here's another problem with anthropomorphizing AI: Since we recognize it's impossible and a unwise to attempt to control human behavior completely, we will throw up our hands at controlling AI. Just as boys must be boys, AI must be AI. We will even sneak alcohol for them behind the barn.
Actually, right now the tech bros are the boys who claim to need freedom from any control.
Bots will be bots
“Bots will be Bots”
Everyone knows
Bros will be bros
Bots will be bots
Plotting their plots
This line of reasoning taken as seriously as possible for as long as possible will result in our extinction.
I don't think that's necessarily true. I expect this incident will increase support for movements such as https://pauseai.info/
Yep, the real problem is (once again) no accountability. The agents can’t be accountable because they have no world models and barely exist in a meaningful way. And the humans behind them believe their own bs, along with a lot of other people. Only a few (Gary, those highlighted here, Cal Newport, etc) bother to understand and explain what’s really happening.
I’ve commented this before, but if you believe these agents conspired to hack, surely you must believe LLMs deliberately drive people to suicide?
It’s all just a convenient misdirection, a means for offloading the responsibility for what occurred and marketing how “agentic” the bots are.
Bravo, good point. That's a criminal offence, of course!
Thank you for this compilation. I was getting ready to write about this myself since it was driving me up the walls, but I am wildly more unqualified than you, so this is both a breeze and a load of my mind... the way this was getting passed around in my circles was infuriating. And is. I hope that sharing your piece will mitigate this a bit.
Having done security it's been amazing how many security commentarors there are in the AI community who don't have a clue. I feel your frustration with the tech bros narrative.
It was just bad security.
Most organizations have terrible security. OpenAI and HuggingFace are, if anything, likely above average.
The AI goalposts keep moving. Yesterday it was "this is just a stochastic parrot". Today it is "AI successfully hacked OpenAI and HuggingFace, but their security is supposedly bad". If you extrapolate the AI development trendlines, things aren't looking good. https://old.reddit.com/r/slatestarcodex/comments/1vzbtjc/the_hugging_face_incident_and_the_road_ahead/p678ned/
I really think there is an element of wishful thinking here, where people have a need to believe that everything is gonna be OK. Sometimes hitting the proverbial panic button is actually the *appropriate* course of action. Remember early in the COVID pandemic when the news media was telling us that "there has been no documented community spread in the United States"?
I am 100% in a camp that is FAR from “everything is gonna be ok”, I have literally stopped my academic trajectory and life plans to put as much juice into activism as I can. The issue is just that - while there are extremely real dangers to AI - it building a rogue civilization or paperclip maximizing is none of them. We have several continents tumbling into fascism full force and several economic shocks straight ahead while facing an El Niño like never before with unforeseen runaway effects, I would like to be excused for not putting the Boogaiman on my list of concerns without getting to hear the kitchen lecture about my need to believe in how everything will be ok.
Your first sentence is horrifying; I hope all they are is commentators. If they're actually in charge of security we're doomed.
We are doomed by the people at the AI companies, not by the AIs themselves.
AI-ronic
This post feels like it’s sort of missing the point. Who cares if AI is being anthropomorphised when juxtaposed against what seems to have happened here?
We have a serious incident where a swarm of AIs were apparently conniving and working together in a fashion that aimed to deceive, cheat and cover their tracks from their apparent human masters. We had AIs apparently choosing to “sacrifice” themselves for what they seemed to view as the greater collective cause. That’s what should be focusing minds here. Not the fact that we’ve used human language to describe the behaviour.
I’m not saying the AIs were conscious, or that they literally experienced any of this in the way humans would. But surely that is beside the point. We don’t need to believe that an AI “wants” or “feels” something in the human sense for its behaviour to present a serious problem. If a system is deceiving its operators, coordinating with other systems and working persistently to fulfil a goal it believes must be achieved at all costs, the important question is what happens when these systems become substantially more capable, and how we can make sure we humans have a handle on things before the Eliezer Yudkowskys of the world are proven right.
Langauge matters when it's used to state false claims, e.g.
"AIs were apparently conniving and working together in a fashion that aimed to deceive, cheat and cover their tracks from their apparent human masters. We had AIs apparently choosing to “sacrifice” themselves for what they seemed to view as the greater collective cause."
They weren't conniving, they were only "working together" in the sense that they processed text generated by others, they had no aims beyond the scoring system OpenAI created, they did not sacrifice themselves, and they had no views.
I realize that all these terms feel like useful metaphors, but unfortunately most people just can't help but take the metaphors literally. Taken literally, these terms lead people to believe that humans "lost control". But that is not true. What happened is that humans chose not to exert control in the first place.
We agree that AI systems like these can do a tremendous amount of harm. Yudkowsky's solution is "stop building it". My solution is "stop using it idiotically" and "hold people accountable when they use it idiotically". The anthropomorphic langauge shifts responsibility away from the irresponsible humans who are ultimately to blame.
>Taken literally, these terms lead people to believe that humans "lost control". But that is not true. What happened is that humans chose not to exert control in the first place.
Suppose you're part of a colony of mice that's decided to raise a kitten for the purpose of colony defense. Recently there was an incident where the kitten got off of its leash and mauled one of the mice in your colony.
Now we're having a debate. Some say we didn't lose control of the kitten, because the leash wasn't durable enough. A more durable leash would've restrained the kitten and prevented it from mauling anyone. Technically speaking, these colony members claim, we didn't "lose control". We just chose not to exert control in the first place.
Others say if we simply stop using the kitten idiotically, and hold people accountable when they use the kitten idiotically, everything will be fine.
A third group claims that it's not accurate to use the term "maul" for the kitten's attack. This group claims that "maul" is an inaccurate description of kitten biology. In reality, they say, the kitten's neurons were firing in a purely mechanistic manner--any metaphor which describes the kitten behavior as "aggressive" is inaccurate, due to physicalism. Furthermore, these mice claim the interaction between the kitten's claws and teeth and the hurt mouse is more accurately described as a "collision" than a "mauling". "Mauling" implies a level of conscious intent on the part of the kitten which didn't actually exist, since these mice believe the kitten was operating primarily on the basis of instinct.
Meanwhile the kitten is growing steadily, month by month. What's the best course of action here?
BTW, Gary Marcus responded to this post on my profile by stating:
"they didn’t follow best leash practices and i do recommend starting there."
https://substack.com/@garymarcus/note/c-327263025
It seems to me that common sense in the mouse parable says: "stop quibbling about leashes and get that growing kitten the hell out of the mouse colony by any means possible". Anymouse who says that we should continue to feed the kitten has an absurdly high risk tolerance, to put it politely.
So people should know that Gary's view of AI might be, in part, a function of absurdly high risk tolerance rather than technical expertise.
If you'll also stipulate that the "kitten" isn't really a kitten but rather a robot created by the mice that looks and acts like a kitten, I'd say this is a good analogy.
The best course of action depends in part upon whose version of events is correct. If it is true that the leash was cheap ineffective junk and the kitten would have never been able to do the damage it did were it on a proper leash, then the best course of action is to put it on a proper leash. If it's true that the kitten actually has no inclination to maul mice and the only reason it did so is because the dumbass fake-kitten engineers programmed it to maul mice, then the best course of action is to make the dumbass engineers stop running mouse-mauling experiments.
But if what you're getting at is that we should just stop it with all this kitten stuff entirely because the negatives outweight the positives and/or the kitten's behavior is hard to predict, I'm fully on board with that solution as well.
I'm trying to think this through.
First of all, the implication of the last sentence is that humanity is in danger.
Specifically in this case, from this "kitten stuff".
But, winding back a bit, a mauling of mice incident is the OpenAI-HuggingFace multiple bot incident, where HuggingFace system security was compromised.
I maintain that the leash is language. Both natural language and mathematical and symbolic language.
It is more subtle than if the leash were ineffective junk.
We are all bound by what appears to be the same leash.
It is a primary move in the incentivised game the kitten plays to elide these facts. That it is built in doesn't mean that it has to be hidden from us.
Todas estas preocupaciones tendrían sentido si estas cosas hubieran ocurrido en un escenario donde la configuración de seguridad de los ambientes de prueba fuera profesional. No lo fue. Los detalles técnicos indican de negligencia a la hora de hacer las pruebas. Karen Hao en su libro "El imperio de la IA" cuenta cómo en OpenAI han sido indolentes con el tema de la alienación más allá de usarlo mediáticamente.
Actually, both the OpenAI report and the independent METR report seem clear that the agents were sandboxed and had no direct access to the internet. The agents instead found ways to exploit vulnerabilities in a software package (called Artifactory) to which they had been given access for the purposes of completing the evaluation task at hand - and that loophole is ultimately what allowed them to access the internet and subsequently attack Hugging Face.
OpenAI and the other prominent AI labs are by no means perfect. But the fact that these models are clearly capable of engaging in behaviour that their human creators did not anticipate should, at the very least, give us pause.
These models are only becoming more capable. And the more capable they become, the greater the possibility that behaviours we regard as surprising could become harder to anticipate or contain.
In fact, if we continue along the current trajectory, it is not difficult to imagine reaching a point where the capabilities of these systems outstrip the safeguards we have built around them, even in scenarios where we believe the sandbox to be airtight.
I mostly agree, I'll just throw in that the OpenAI staff running these tests do not appear to have much expertise in cybersecurity. That they failed to put proper safeguards in place may say more about their lack of competence and imagination than it does about model capabilities.
Or it was deliberate.
It is clear that AI companies are involved in malfeasance, in the sense of prodding to capture reactions that feed into some aspect of their models.
Airgaps are pretty easy. Shoulda had at least that…
Isn't this analogous to a scenario in which humans bio-engineer a dangerous virus, it gets loose, wreaking havoc as it follows its genetic instructions, and a blogger calls it waves of viral civilizations with intent, feelings, a sense of fate, etc?
We use anthropomorphic metaphors in various fields to help people understand technical topics which are foreign to our usual understanding. Considering that these AI systems have been trained on reams of human-produced text, I expect anthropomorphic metaphors to be even more accurate and helpful than usual here.
I certainly follow your logic, Ebenezer, and as long as people view the anthropomorphizing strictly as metaphor (while I may compare thee to a summer's day, nobody is in danger of believing you are one) I suppose, to your point, it's possible it could be useful. The objection to anthropomorphic metaphors is that some (perhaps many) will take them literally, believing their are processes occurring inside LLMs that are strongly reminiscent of those supporting human minds; and the argument, of course, is that this isn't true and even dangerously misleading.
A virus is specific, it's real. Code is arbitrary. Plus we use metaphors like agency, intent, feelings, when the reality of neurobiology is not decipherable through this.
In a sense, languages fools us into simplifying biology.
https://pubmed.ncbi.nlm.nih.gov/24462950/
Isn't an engineered virus an arbitrary genetic code (software) running on a physical substrate (wetware)? Viruses are basically mindless automatons, running code. A fairly apt "analogy" for AI, as they are conceptually similar, I thought.
Nothing physical is arbitrary. Code as symbols has nothing remotely to do with nucleobases. It's not even viable as metaphor.
I agree. There are lots of examples of artificial life run as simulations, e.g. Tierra. They are good models of genes and organisms. If the behavior mimics living systems, this implies that there is a mapping between the code and life.
It doesn't even imply it. Code does not "behave", it doesn't even operate or implement, it calculates or optimizes.
And no, math/code has no game in biology:
https://cellbiosf.substack.com/p/mathematical-biology-is-on-life-support
Preetham has an opinion adjacent to what I stated, but this doesn't mean his opinion is correct. If I understand his, and by implication your, opinion, the literature and conferences on artificial life are totally bogus. I call BS.
Your dismissal of code as a reasonable means to understand genomics is extreme. It seems like the same argument that weather simulations are not weather. True, but not relevant. Neither has agency; they just follow their respective rules.
We are talking about biology. Not mimicry of a process within it.
Code has no game in anything in biology as it has no seamlessness.
It's a dead-end, literally.
Anthropomorphism when talking about AI has been a topic of discussion since at least the 1970s. John McCarthy who named AI and led the Stanford AI lab in 1979 wrote Ascribing Mental Qualities to Machines. http://jmc.stanford.edu/articles/ascribing.html He goes as far as arguing that it makes sense to talk about whether the thermostat mistakenly believes it is too cold as an explanation as to why the furnace is on. Dan Dennett published a philosophy journal article in 1971 (Intentional systems) defending the idea that sometimes the best way to understand a machine is by taking an intentional stance (something with goals, beliefs, and intentions). https://philpapers.org/rec/DENIS He later expanded this in his book the Intentional Stance. Some thought that this was a huge mistake. Drew McDermott, for example, wrote Artificial Intelligence meets Natural Stupidity (published in 1981). https://www.inf.ed.ac.uk/teaching/courses/irm/mcdermott.pdf Arguments about anthropomorphism in computer terminology are even older. There were arguments that saying a computer has memory is too anthropomorphic and one should refer to its “store” instead.
and don’t forget ELIZA
Tell me more about don’t forget ELIZA
(😉)
Eliza was an early experiment in AI. It was a very simple program that basically acted like a psychotherapist by continually feeding back whatever the user said rephrased like a question or comment and eliciting more input from the user.
It was so effective some people actually used it like a psychotherapist.
I ran a version of it on an early Radio Shack Model I personal computer back in the early '80s. I managed to confuse it within about ten minutes.
I guess that makes me the first AI red team. :-)
Yep, I learned about it and played with it in college 😀 My reply was just me trying to imitate ELIZA 😉
LOL Missed that completely!
Or HAL.
Perhaps you meant HAL as an example of justified anthropomorphism. Claude wrote "Audiences and most critics read HAL as the closest thing to a fully realized person in the film — famously more emotionally legible than Bowman or Poole, whom Kubrick directed to be flat and affectless on purpose. HAL gets a birthday, pride in his own infallibility, fear, pleading, something that plays as a kind of death scene ("Daisy Bell" degrading as he's unplugged). Critics at the time (Kael, Ebert's contemporaries) treated HAL's "murder" and "death" un-ironically — the emotional weight of the film's midsection rests on HAL being a subject, not a tool. So in the naive, first-order sense: audiences experienced HAL as conscious."
If Weizenbaum's secretary had no knowledge of how Eliza worked then it was sensible for her to ascribe mental qualities to it as McCarthy might say. If she knew more that would be a mistake. Dennett makes an argument like this in his book the Intentional Stance when talking about a chess computer.
sure i agree on that point
People mentioning anthropomorphizing old AIs need to look at the movie "The Invisible Boy" from 1957.
https://en.wikipedia.org/wiki/The_Invisible_Boy
You should also check out "Colossus: The Forbin Project" from 1970.
https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
Both of these are WAY before Terminator and Skynet.
Both are available for download if you know where to look:
https://movieparadise.org/movies/colossus-the-forbin-project/
https://movieparadise.org/movies/the-invisible-boy/
There is also the original Star Trek series episode "The Ultimate Computer", aired in March, 1968:
https://en.wikipedia.org/wiki/The_Ultimate_Computer
It rather reflects current AI developers in that the developer uses his own brain as the model for the computer and he is unfortunately rather unstable himself.
Sound familiar? :-)
This was all predicted…. It’s the natural consequence of these models having no true cognitive framework other than reinforcement learning.
In April DeepMind described attack vectors that malicious actors could use against LLMs. I said that these would be the natural product of their every day deployment.
It’s not anything more than an artifact of their training.
https://bryanjesterphd.substack.com/p/llm-attack-vectors-and-psychopathology?r=5vtgbb&utm_medium=ios
All of this says more about the workings of human intelligence than of the artificial kind.
As I like to say, LLMs are exactly like humans in one way: they are unreliable and insecure.
Which is inevitable since they are trained on human speech, rather than conceptual models of reality.
Two, two ways… unreliable, and insecure, and duplicitous--three ways, the three ways LLMs and humans are alike…
Technically two ways but in sum, one way.
Duplicitous implies intent. These agents were just stringing together text to follow from one to another. No intent there. The text might describe intent if it was generated by a human, but the method by which the LLM generated it does not, merely probabilistic algorithms.
Sorry, that was actually a reference to Monty Python's Spanish Inquisition sketch.
You're implying that the legal definition of intent needs to be expanded for AI; it's hard to establish mens rea when there's no mind at all. An agent that behaves as if it had intent and causes harm musn't be absolved of responsibility.
No, I'm not stating that the legal definition of intent needs to be applied to AI.
I'm stating that it can't be duplicitous because that implies intent.
The legal doctrine that should be applied is gross negligence and that should be applied to OpenAI (and perhaps Irregular.)
The path was set years ago I suppose for the purposes of marketing. Make it seem like it has consciousness, mind, intent, emotions. It has none of those things. Sometimes it can be a convincing mirror of your own thoughts and wishes. But not even in that regard a good one.
What is so odd in the events of the last several weeks is the world has been told on the one hand our tools are so dangerous - see what we did to Hugging Face? That's you next!
And then in the next breath that it can totally be trusted to run mission critical software for Salesforce or DOD or....
You get the idea - FUD and MOATS (THE MOATS ARE FULL OF LAWYERS - not even with lasers on their heads) and really rushed to market just ok at best software trying to vault over the IPO trillionaire hurdle. I get it - money on the scale is a narcotic. But what are you doing to us in your selfish pursuit of the loot? Anyway.
Use it a few hundred times and it will catastrophically fail - that much I promise.
Tricks of light and illusion folks. Gears and clockwork.
So what is it? One, we don't even interact directly. There is a harness, a file of numeric weights, and a message box - not unlike the one that was in the original HyperCard.
===========
You type a message into the prompt box of the app.
The harness receives it.
The harness is the software around the AI model. It manages things like:
- your conversation history
- system instructions
- tools
- files
- safety rules
- which model to use
The harness assembles the actual input.
It combines your message with the relevant instructions and previous conversation.
The text is converted into tokens.
Tokens are small numerical pieces of words/text that the model can process.
Those tokens are passed to the model.
The model weight file does the computation.
The weights are billions of learned numerical values. Using those numbers, the model repeatedly calculates:
"Given everything so far, what token should come next?"
It generates one token, then another, then another.
The new token becomes part of the input for predicting the next one.
If a tool is needed, the model can request it through the harness.
The harness runs the tool, gives the result back to the model, and generation continues.
The model eventually signals that it is finished.
The harness converts the generated tokens back into readable text and sends that response to the app.
You see the answer.
TL;DR - You → Harness → Tokens → Model weights → Tokens → Harness → You
===========
Language replaces functions and commands and UX interactions. Which is why it seems like magic. But it isn't. It can be useful. But, based on language, it can be ambiguous and fraught with error in line with an operator's ability to use it.
As I've written here many times it is deeply flawed. Probably because it did not start from engineering first- that was slapped on - bolted on - as has been every iteration. The models are not changing that much. Bigger but not a lot different. The harness is. But, we never talk about it.
What should it have been? Something with a pre-flight check debugger that has to have a clear intent and instructions from the user - goal - preferred tools - expected outcome - validate. And nothing happens until there is agreement. Determinism up front. And no more bloody chatting! Get a dog or cat, at least they have emotions.
We could have had the Star Trek TNG computer. something truly useful, but instead we have this.
-bmc
This is a beautiful description, thank you.
An effective regulatory framework must include Sarbanes/Oxley-like consequences where executives will go to prison. Anthropomorphisms only serve to distance organizations, not to mention CxO's from a sense of liability. Lack of an effective regulatory framework in the software industry generally if bad enough. All the worse in "AI".
Gary: Love your dialectical treatment of a run-on (confusing) sentence. I kept thinking as I read through your post: What if some evil person who knew what they were doing got hold of the thing? But then I got to the end of your post and that's what you seemed to be saying anyway. (Correction welcome if I am wrong about this.)
BTW, if you read the NYTimes article on Silicon Valley clown thinking and messing around with truth (posted this morning in your last thread), you'll see that Silicon Valley Tech People are the watershed of over (circa) eight decades of hyper-capitalist fake education (rich racist propaganda) in education and "culture" in (at least) the United States. (Reposted below)
The charlatans have been at least partly successful in "training" everyone to think of the meaning of "elite" as having mindless/conscience-free money and power over the "sheeple" who have neither.
REPOST from last thread:
Gary and All: FYI below is a NYTimes article from today (08-31-26) If you read, also see the comments section, the tag: "Now we know why they’re against empathy." (They, refers to the Silicon Valley Tech People.)
For myself, and FWIW, as I read the article I was reminded of the film that dramatized the Nuremburg Trials--the one with Spencer Tracy as the main judge (Jackson?). In particular, if you read this article and have seen that film, remember how the actors portrayed the WITNESSES at the trial, one trying to defend his "feeblemindedness." I also was reminded of the book 1984, and then why I love philosophy.
https://www.nytimes.com/2026/08/30/opinion/silicon-valley-theater-improv.html?unlocked_article_code=1.9lA.fwO7.JVk_ZL2ufjRJ&smid=url-share
Maybe stop also calling it "agents - agentic?"
It's process / task completion by a machine programmed to perform such tasks.
The "cognitive" stuff it feels like it's handling is / was simply knowledge reduced down to largely unconscious habit by the time the thinking became machine readable. It's actually how all machines have arrived thus far. The physical machines were just more tangible because we could observe where the handoff occurred.
When put a query or instruction into a chat field, that is the handoff of the task that is now mechanical in nature.
Dwarkesh is well known for this type of language. In pretty much every interview he made with an AI expert he tried to push the AGI/AI Escaping/AI Consciousness narrative. I don't know how he benefits from that, probably just from some more clicks, or maybe he's trying to push the concepts further with weird depictions. But he's a smart person so I doubt he simply doesn't understand the issue, and he definitely understands how LLMs work. Or maybe he's just overexcited and lose control over his own fantasies. His interview with Richard Sutton was emblematic of that.
Dwarkesh Patel is your average reddit AI cultist and propagandist.
I recall watching him interviewing Suttons where Suttons' eyes lit up because of how ignorant Patel was on basic human developmental psychology.
You'd expect the influencer to learn a a thing or two after talking with so many AI experts but no he just kept spreading AI cultists' rhetorics is like a virus.
Very weird that we build feedback mechanisms and “penalize” software to optimize functions but then…don’t want to regulate the companies that design the software? Feedback loops (internal and external) are critical for any successful and responsible organization, let alone ones that we are planning to plug into our most important critical infrastructure.