What is it exactly that they are doing? I'd assume more, I can't remember the term atm, tentative token prediction and branch comparison before final token selection. A memory:precision tradeoff vs. "chain of thought" essentially.
CoT monitoring is not really that useful in the first place. There's this delusional belief that you're seeing some sort of accurate reporting on internal model state, when actually it's a context refinement hack. There's no way to know what the model is "thinking" because it isn't thinking in the first place, and you can't with any accuracy predict what it will predict next based on prior context.
If you want security, stop the model from *doing* things it shouldn't do. The tokens it predicts are irrelevant to security if they can't result in real world damage (other than hurt feelings anyway) because you properly constrained it to only the things which it should do, instead of foolishly hoping you could coerce it into avoiding doing bad things through context manipulation.
That’s in line with a lot of what I’ve seen from security professionals. Context manipulation is useless and nobody in their security group seems to know how to properly isolate these test environments.
Or in other words: the gang that can’t shoot straight is conducting ballistic missile tests, and expecting us to cheer for them when they nearly blow up a hospital.
Well, you know, to restrict access is to necessarily constrain capability, and their IPOs live or die on how many capabilities they can brag about. So of course their sandboxes are sieves.
It's not even a hard problem to solve. Trivially, if you don't want the model to *ever* do anything bad, don't execute any code or tool calls. Boom, perfect security.
But of course you don't want that, because then it's not useful.
So you can take a whitelisting approach and very precisely permit very specific tool calls that are demonstrably, provably harmless. There are ways to do this *within* inference if you control the execution environment (open weight/local models) using techniques like llguidance. Or you can just check a tool call against a whitelist and not execute if the model has generated a bad call.
Or you can be really dumb and try to heuristically blacklist things based on what has gone wrong before. Which is the present industry standard. Which is how you open the door to "hacking".
Somewhere in between the extremes, a reasonable balance of risk vs. utility.
With humans we have some level of trust, but we don't entirely rely on trust. You might be 100% confident that all the people in your office are stable, intelligent, and of good character. You still don't let them wear explosive vests to work. For one thing, because none of the work you need done requires that. But you don't wait until the first suicide bombing to set that policy. LLMs are paradoxically less trustworthy and yet granted more permissivity than humans, for reasons I will never understand.
Forgive me for going slightly off topic with this significant and worrying issue, but have you tried using an AI tool to generate an accompanying infographic to render your vivid concept "a barrel of tits"? Asking for a friend, who's unfamiliar with the term...
They want to make models harder to monitor because the technology has plateaued and they don't want people to find out the "New Epic" model is just a reskin of the model they rolled out 3 months ago
También creo que ésa es la verdadera razón. Ocultan lo que pasa para que pensemos que es demasiado importante para saberlo. En tanto, seguir adelante con la financiación.
One European hook that has not come up in this thread: the Commission designated ChatGPT a Very Large Online Search Engine on 31 August, which puts OpenAI under an annual systemic risk assessment and an independent external audit with a January 2027 deadline. The AI Omnibus that entered into force on 27 July also extended the AI Office's supervisory reach to AI systems integrated into DSA-designated platforms and search engines. Does a deliberate reduction in chain-of-thought legibility fall inside what a DSA auditor can ask about, or does it sit outside that regime entirely?
CoT isn't "thinking" and so isn't a good way to monitor a Russian Roulette machine.
I suspect that they dropped it since CoT blows up the cost of inference without bringing real benefits for predictability of the model behavior.
In a way I think it is better to focus on different ways to deal with AI unpredictability.
I invented a new term today because it was floating inside my head for a whole.
The only way to address AI predictability is by developing and using the concept/strategy of what I call Predictable Algorithmic Shielding or PASh (unless we find a better name).
This is a method by which the LLM is used only to generate PA (classic code) that can be then tested and verified.
This is a direct replacement of CoT but this time a real algorithm not a fake one as the CoT is.
We know that these systems will never be completely trustworthy so let's not be kidding ourselves and start from that knowledge.
Since LLMs can be made relatively good at coding then we can leverage this and use this ability in order to create a predictable way to use LLMs by encasing the LLM in a PASh.
Most important we must replace the current agentic "abomination" with a PA strategy.
That can make the automation using AI actual reliable.
I'd continue but I'll go to sleep and see what tomorrow brings in this arena.
I tend to agree. Any powerful LLM system needs to operate under close human control.
I would suggest that CoT got the flick entirely for cost reasons. OpenAI has a lousy safety culture, as proven by this and the Hugging Face attack. Yes, CoT doesn't help you predict the model's behaviour but it at least lets you check what it did to an extent. Ditching CoT likely isn't a technical trade-off; more probably OpenAI simply don't care.
Great, great idea; when the interpreter hallucinates, any trip into the AI can generate a hallucination, but compiling it (in some way) is obviously a much better architecture.
This limits potential hallucinations to those that are (presumably) caught at compile time.
You can also begin instituting version control, as well as monitoring/logging points. You can set breakpoints. Everything we want.
Is there not a standard on this? There needs to be one: if not to create code, at least to create analogs to .o and .dll files (for all the reasons we have those things).
Splunk would be happy to have such a standard, and imagine the new tools that would flourish.
I have a suggestion: lock all the Sam Altmans in a sandbox and throw away the key.
Why, oh why are these psychopaths allowed to unleash these things on the public??? They should be sorting all this out in private with zero access to the public internet. Instead they are hogging all the money, power, and resources for their own personal enrichment. NO BAILOUTS. Hold them responsible for damages they cause.
You just invented a new noun and adjective. Cheers! But I refuse to concede there's no way out. Never submit to fascists in advance. Gary's certainly not giving up, thank goodness.
Marcus, do you believe their currently displayed CoT is the real one? I remember reading somewhere that is mostly fake anyway, designed to build trust with the user with a facade of transparency.
Having led teams that built AI systems at Amazon and Microsoft, the monitoring and explainability of model-based workflows is either not built in, or it is transformed and difused before it reaches the user. Only sanctioned engineers would have access to the true steps and be able to replay them, in a secure sandboxed environment.
It won't make it failsafe necessarily, but it at least lets you trace the problem (either directly or when you detect a big mismatch between what you wanted and what the LLM gave).
Indeed. I'm not debating the underlying architecture. My point is just that most model providers are likely to add a filter on top of the actual CoT instead of showing the raw steps, even if LLM generated.
I share the concern about losing visibility into chain of thought. If we currently have a useful way to observe how a model is reasoning, making that harder to inspect is obviously something worth taking seriously.
What I’m less convinced by is the idea that this visibility can be the safety boundary itself. More capable models may increasingly reason in ways we cannot reliably inspect, whether because of architecture, training, or simply because our monitoring methods stop scaling with the systems.
If that happens, governance has to survive the loss of visibility. We may not know why the model wants to take an action, but we should still be able to control whether that action is permitted, what systems it can reach, and under what conditions it can proceed.
So yes, monitor the reasoning wherever that remains useful. I just wouldn’t want the safety model to depend on being able to see inside it.
Your post with Zack Korman was absolutely spot on. OpenAI do not appear to be trying to be safe. To the extent that they do it is, almost literally, security theater. Unfortunately they don't have a method to stop the actors once they get off the stage and jump into the crowd.
This latest seems to be at the level of "ravenous bugblatter of Traal" - If I can't see a security breach happening then it didn't happen, so I'm going to make it harder for me to see breaches
At least we can be sure that OpenAI is not planning to reveal less of the thinking of their engineers and company officials, since that is not possible.
People are concerned/interested in whether an AI is "conscious", but what we are seeing in all this is analogous to the problem in humans of normality vs psychopathy. Why and how does someone, seemingly "normal" turn out to do things totally bizarre or malevolent? We can't really understand this in humans, and we certainly can't predict who is going to cross a boundary into a horror event. Now, we are creating software that has the same potential for mischief, and we test it without putting it in the equivalent of a biosafety level 4 facility. A "sandbox" is not the right analogy. "Who knows what evil lurks in the hearts of men..." We don't have a "Shadow".
But you don't need to. When the OpenAI leadership team are in the hot seat in front of a Washington committee or a court, you'll have all the graphics you could ever want.
What is it exactly that they are doing? I'd assume more, I can't remember the term atm, tentative token prediction and branch comparison before final token selection. A memory:precision tradeoff vs. "chain of thought" essentially.
CoT monitoring is not really that useful in the first place. There's this delusional belief that you're seeing some sort of accurate reporting on internal model state, when actually it's a context refinement hack. There's no way to know what the model is "thinking" because it isn't thinking in the first place, and you can't with any accuracy predict what it will predict next based on prior context.
If you want security, stop the model from *doing* things it shouldn't do. The tokens it predicts are irrelevant to security if they can't result in real world damage (other than hurt feelings anyway) because you properly constrained it to only the things which it should do, instead of foolishly hoping you could coerce it into avoiding doing bad things through context manipulation.
That’s in line with a lot of what I’ve seen from security professionals. Context manipulation is useless and nobody in their security group seems to know how to properly isolate these test environments.
Or in other words: the gang that can’t shoot straight is conducting ballistic missile tests, and expecting us to cheer for them when they nearly blow up a hospital.
Well, you know, to restrict access is to necessarily constrain capability, and their IPOs live or die on how many capabilities they can brag about. So of course their sandboxes are sieves.
It's not even a hard problem to solve. Trivially, if you don't want the model to *ever* do anything bad, don't execute any code or tool calls. Boom, perfect security.
But of course you don't want that, because then it's not useful.
So you can take a whitelisting approach and very precisely permit very specific tool calls that are demonstrably, provably harmless. There are ways to do this *within* inference if you control the execution environment (open weight/local models) using techniques like llguidance. Or you can just check a tool call against a whitelist and not execute if the model has generated a bad call.
Or you can be really dumb and try to heuristically blacklist things based on what has gone wrong before. Which is the present industry standard. Which is how you open the door to "hacking".
Somewhere in between the extremes, a reasonable balance of risk vs. utility.
With humans we have some level of trust, but we don't entirely rely on trust. You might be 100% confident that all the people in your office are stable, intelligent, and of good character. You still don't let them wear explosive vests to work. For one thing, because none of the work you need done requires that. But you don't wait until the first suicide bombing to set that policy. LLMs are paradoxically less trustworthy and yet granted more permissivity than humans, for reasons I will never understand.
When everyone in power making the decisions is heavily invested (to the tune of trillions overall), permissiveness is virtually guaranteed.
You said there would be a black swan event that would scupper the AI craze. Here it comes.
"Unpredictable, unalignable LLM knocks over major bank/utility because OpenAI needed some key to jangle".
indeed that is exactly what I said
Thing is, I actually worked for a major utility. Basically any fault in most of these systems can at least be *managed* until it's fixed.
EXCEPT for a central computer/control centre knock-out. That will always cause mayhem.
So what do OpenAI do? Make it really easy for an attacker to do just that! All while your back is turned because you can't monitor the model!
This could have been stage-managed to boost Anthropic's IPO. OpenAI are going to look a barrel of tits when this goes wrong.
Forgive me for going slightly off topic with this significant and worrying issue, but have you tried using an AI tool to generate an accompanying infographic to render your vivid concept "a barrel of tits"? Asking for a friend, who's unfamiliar with the term...
https://garymarcus.substack.com/p/red-alert-openai-is-poised-to-cross/comment/327953655?utm_source=share&utm_medium=android&r=5pqhu1
OpenAI are going to run out of money and cause financial mayhem as all the vendor financing and RPO dominos start to fall.
Is this related to 'going tits up'?
It’s quite clear we are quite fucked
And yet no one is stopping this, much as people like Gary are trying to warn them.
They want to make models harder to monitor because the technology has plateaued and they don't want people to find out the "New Epic" model is just a reskin of the model they rolled out 3 months ago
También creo que ésa es la verdadera razón. Ocultan lo que pasa para que pensemos que es demasiado importante para saberlo. En tanto, seguir adelante con la financiación.
One European hook that has not come up in this thread: the Commission designated ChatGPT a Very Large Online Search Engine on 31 August, which puts OpenAI under an annual systemic risk assessment and an independent external audit with a January 2027 deadline. The AI Omnibus that entered into force on 27 July also extended the AI Office's supervisory reach to AI systems integrated into DSA-designated platforms and search engines. Does a deliberate reduction in chain-of-thought legibility fall inside what a DSA auditor can ask about, or does it sit outside that regime entirely?
CoT isn't "thinking" and so isn't a good way to monitor a Russian Roulette machine.
I suspect that they dropped it since CoT blows up the cost of inference without bringing real benefits for predictability of the model behavior.
In a way I think it is better to focus on different ways to deal with AI unpredictability.
I invented a new term today because it was floating inside my head for a whole.
The only way to address AI predictability is by developing and using the concept/strategy of what I call Predictable Algorithmic Shielding or PASh (unless we find a better name).
This is a method by which the LLM is used only to generate PA (classic code) that can be then tested and verified.
This is a direct replacement of CoT but this time a real algorithm not a fake one as the CoT is.
We know that these systems will never be completely trustworthy so let's not be kidding ourselves and start from that knowledge.
Since LLMs can be made relatively good at coding then we can leverage this and use this ability in order to create a predictable way to use LLMs by encasing the LLM in a PASh.
Most important we must replace the current agentic "abomination" with a PA strategy.
That can make the automation using AI actual reliable.
I'd continue but I'll go to sleep and see what tomorrow brings in this arena.
Just MHO...
I tend to agree. Any powerful LLM system needs to operate under close human control.
I would suggest that CoT got the flick entirely for cost reasons. OpenAI has a lousy safety culture, as proven by this and the Hugging Face attack. Yes, CoT doesn't help you predict the model's behaviour but it at least lets you check what it did to an extent. Ditching CoT likely isn't a technical trade-off; more probably OpenAI simply don't care.
(PASh is a good acronym, by the way.)
Great, great idea; when the interpreter hallucinates, any trip into the AI can generate a hallucination, but compiling it (in some way) is obviously a much better architecture.
This limits potential hallucinations to those that are (presumably) caught at compile time.
You can also begin instituting version control, as well as monitoring/logging points. You can set breakpoints. Everything we want.
Is there not a standard on this? There needs to be one: if not to create code, at least to create analogs to .o and .dll files (for all the reasons we have those things).
Splunk would be happy to have such a standard, and imagine the new tools that would flourish.
I have a suggestion: lock all the Sam Altmans in a sandbox and throw away the key.
Why, oh why are these psychopaths allowed to unleash these things on the public??? They should be sorting all this out in private with zero access to the public internet. Instead they are hogging all the money, power, and resources for their own personal enrichment. NO BAILOUTS. Hold them responsible for damages they cause.
Unfortunately, we (the public) are all locked in a Sambox with no way out.
You just invented a new noun and adjective. Cheers! But I refuse to concede there's no way out. Never submit to fascists in advance. Gary's certainly not giving up, thank goodness.
“Samboxed”
Locked in a Sambox
With no way out
An LLM black box
Without a doubt
Why? My non-expert opinion is we are living in a Pathocracy, and Altman-types are nearing total control.
"Nearing" is the operative word here. Time to rise up and take his and their power away, not sleepwalk off a cliff.
Marcus, do you believe their currently displayed CoT is the real one? I remember reading somewhere that is mostly fake anyway, designed to build trust with the user with a facade of transparency.
Having led teams that built AI systems at Amazon and Microsoft, the monitoring and explainability of model-based workflows is either not built in, or it is transformed and difused before it reaches the user. Only sanctioned engineers would have access to the true steps and be able to replay them, in a secure sandboxed environment.
Per “Rao’s work that I cited i think it’s unreliable, correlated better than nothing but not a direct report of what goes on under the hood.
Like the chain of command.
It won't make it failsafe necessarily, but it at least lets you trace the problem (either directly or when you detect a big mismatch between what you wanted and what the LLM gave).
It's imperfect, but it can be useful. CoT is generated by the LLM itself (I believe), so it can definitely be hallucinated.
It's a pragmatic solution. It's certainly not perfect, but it at least does some of the job and it's better than the previous alternative - nothing.
Indeed. I'm not debating the underlying architecture. My point is just that most model providers are likely to add a filter on top of the actual CoT instead of showing the raw steps, even if LLM generated.
I'm saying this from hand experience.
Replied here
https://garymarcus.substack.com/p/red-alert-openai-is-poised-to-cross/comment/327826958?utm_source=share&utm_medium=android&r=5pqhu1
Useless Substack app
I share the concern about losing visibility into chain of thought. If we currently have a useful way to observe how a model is reasoning, making that harder to inspect is obviously something worth taking seriously.
What I’m less convinced by is the idea that this visibility can be the safety boundary itself. More capable models may increasingly reason in ways we cannot reliably inspect, whether because of architecture, training, or simply because our monitoring methods stop scaling with the systems.
If that happens, governance has to survive the loss of visibility. We may not know why the model wants to take an action, but we should still be able to control whether that action is permitted, what systems it can reach, and under what conditions it can proceed.
So yes, monitor the reasoning wherever that remains useful. I just wouldn’t want the safety model to depend on being able to see inside it.
Your post with Zack Korman was absolutely spot on. OpenAI do not appear to be trying to be safe. To the extent that they do it is, almost literally, security theater. Unfortunately they don't have a method to stop the actors once they get off the stage and jump into the crowd.
This latest seems to be at the level of "ravenous bugblatter of Traal" - If I can't see a security breach happening then it didn't happen, so I'm going to make it harder for me to see breaches
Largely marketing hype
“OpenAI seems to be violating one of the few red lines that exist in the AI industry” — Steven Adler
Actually, the only red line that exists in the AI industry is the one that underlines financial loss.
At least we can be sure that OpenAI is not planning to reveal less of the thinking of their engineers and company officials, since that is not possible.
“Researchers are concerned bc it obscures a model’s thinking process” —Stephanie Palazzolo
Is this one of those Zen Bottist Koans?
How is it possible to obscure something that is not there?
Selling an invisibility cloak to the Invisible Man, is now the true mark of a master salesman. (And I do mean mark.)
“The AInvisible Man
AInvisible “man”
Is just a bot
But risible stan
AIpplaudes alot
People are concerned/interested in whether an AI is "conscious", but what we are seeing in all this is analogous to the problem in humans of normality vs psychopathy. Why and how does someone, seemingly "normal" turn out to do things totally bizarre or malevolent? We can't really understand this in humans, and we certainly can't predict who is going to cross a boundary into a horror event. Now, we are creating software that has the same potential for mischief, and we test it without putting it in the equivalent of a biosafety level 4 facility. A "sandbox" is not the right analogy. "Who knows what evil lurks in the hearts of men..." We don't have a "Shadow".
I haven't tried XD
But you don't need to. When the OpenAI leadership team are in the hot seat in front of a Washington committee or a court, you'll have all the graphics you could ever want.
Touché 🎯