"If you want to know how something works, you look at the code."
The problem with open source LLMs is that the code is the easy part to verify. However, the behaviour of the LLM is fundamentally based on the trained weights, which cannot be verified at all, however many people gaze at the values of billions of trillions of weights.
This has been the problem with all the pattern finding analytics algorithms since about 2002 and Big Data. It is almost trivial to check the code but the behaviour depends on the learned values and weights which cannot be validated at all.
They can't expose their training sets without exposing themselves to fair use lawsuits. Speaking of which, how stupid is the US court system that judges think fair use includes taking every single bit of a company's data and using it to create a product that directly competes with that data for customers?
And how stupid are companies paying for the privilege of training the systems that will replicate their processes and replace their services (many, not all, of course)? Like all the manufacturers that outsourced to China. Either happily trading a few years of turbo profits or grudgingly being forced into it by price pressure and ending up dead after handing over IP and training the lower-cost competition.
So far just that one, as far as I know, but the rest are coming. There is no universe where fair use can be construed to mean, "use every last drop of a company's content in order to create a product that will steal customers away from that content." That is the opposite of fair use. I'm astonished that there haven't been more cases yet other than Bartz.
Bartz is a fiction author who did not allege that any of their content ended up in the Claude models. The case was only about piracy of the books used for training the model to write like a human.
lol I do hope we're saner than to think AI is THE third exception to the rule of law, alongside such eternal edge cases as romance and literal war. AI is a product/service. It should be nailed to the ground like Gulliver among the Lilliputians. I'll be there with a nail gun to help.
I'll take your word that Zuck knows the difference between "open source" and "open weight". However, the first provided tweet on releasing the Muse code does not say it is either open source or open weight. The 2nd example, commenting on Nadella's tweet, could be interpreted as conflating the two, rather than "bait and switch." IOW, I read it as that it wasn't deliberate, but rather confusion.
Lovely to read your post on this, Gary~ This issue was itching at me and my colleagues a while back. So we wrote a paper on this topic not too long ago about how how the media discusses openness in AI (incl. how NYT have been using the terms) https://doi.org/10.1609/aies.v8i2.36688
TLDR, it's not just the NYTimes that get it wrong. Fun fact, there's no easy translation for "open source" in French and they use the English term.
And there's a lot more to the "openness" concept than just mis-labeling things as being open source (if you care more academic discussions, here's a taxonomy of openness we developed as well https://doi.org/10.1145/3715275.3732087).
Open weight is modern freeware. You can have it for free, can run it, and if you're a real nut, meddle with it with the LLM equivalent of a hex editor. The word "open" is smuggling a lot of unearned associations here, by design.
Thanks for the excellent summary. It leaves me wondering what these AI companies think people ARE supposed to do with their open-weight models. Perhaps they are hoping the community will invent new techniques for working with them. For example, perhaps we can train a model on common law only and then subtract it from one of their models, resulting in a new model that focuses on statutory law only. (Commenters, please don't tell me why this won't work. You'd be missing my point.) Are AI companies releasing these open-weight models solely in order to fool people into thinking they are actually open source?
Sadly this, basic, level of accuracy and understanding seems to be standard for modern mainstream media. And across all fields. It’s problematic to say the least.
This has been a problem for a long time in MSM. Science reporters with actual knowledge were replaced by those without it. These days, blogs by scientists (or other domain specialists) are often the best place to understand new discoveries or technology.
A personal case in point about expertise. On this blog or another, a comment I made was strongly corrected by someone with greater understanding of post-training than I. It proved a good "teachable moment" for me, which corrected my understanding, for which I am grateful. These expert blogs can be so much better than MSM reporting, and comments with dialogue can also be very instructive to add information, clarify points, and correct misconceptions.
It's not just this one article. Many articles on AI from the NYTimes are like this where they report on something but don't do enough to understand the topic. By not understanding it well enough, they also aren't asking the critical questions to challenge claims being made by AI companies, for example, often taking what they say at face value or misinterpreting it.
Another thing I've noticed is that whenever they have an article criticizing AI, there is almost always a paragraph in there that says something like:
"Don't let this criticism detract from the amazing progress being made... AI is revolutionizing XYZ". Why can't a criticism just be a criticism without softening it?
Question for those more intelligent than me. I am struck by the prospect that Chinese models should be FAR more cost-competitive than American models, and I only see that disparity growing larger in the future. Because...
- They already have cheaper power than the US
- To my understanding, they are bringing an order of magnitude more power online in the next decade than the US (especially nuclear & solar)
- They can build stuff (i.e., data centers) far faster & cheaper than the US, and without NIMBY-ism concerns
- Subsidies from the CCP
This would seem to position companies like DeepSeek and Kimi very well, to be able to offer models that are, e.g., ~80-95% as powerful as US models, but at a fraction of the cost.
HOWEVER, I understand that their models are open weight, but NOT open source.
The question, then: fundamentally, to what extent can we really expect to see western businesses feel comfortable deploying these AI tools into their infrastructure in a meaningful way, absent the ability to truly audit the source code of these models coming from these Chinese firms? Or will distrust rule the day and prevent these Chinese models from ever taking off in a big way?
Still, labeling will not change the economics that make these models so attractive. Those economics are Beijing’s strategy. Chinese A.I. is both inexpensive and good. The United States still has the best A.I. models, but across many prominent benchmarks, the top Chinese A.I. models are only several months behind. Moonshot’s newest model, released in July, appears to closely trail the best Claude and ChatGPT models, although more assessments are still coming in. Even if it is well behind, it is free, like most leading Chinese A.I. models and unlike the leading American ones.
If Washington wants the world to use American A.I. models, it needs to match China on price and quality. The U.S. government should encourage American companies to cut prices on their older A.I. models that are as good as China’s. It should also support the development of safe, open alternatives to China’s models.
One would hope that when faced with two products of roughly the same price and quality, most businesses and consumers will choose the option that was not built to serve a foreign government. Some American companies may reduce prices on their own to prevent China from capturing the market. But they should not have to. Chinese A.I. is inexpensive because its government makes it so.
Washington should deploy its full arsenal to level the playing field, financing American A.I. infrastructure abroad and lowering costs at home by lightening regulatory burdens and building energy infrastructure. The government should keep exposing these models’ bias and should fund research into preventing Chinese A.I. companies from copying American innovations (primarily through a practice known as distillation).
Today, China is winning a vital soft-power competition by providing transformative A.I. to the world. President Xi Jinping gave a speech on July 17 at the World A.I. Conference in Shanghai that presented China as the noble and generous friend every nation would want. He described China as committed to open A.I., countering speculation among some experts that China might stop allowing the free downloading of its models. He also said his government was committed to international cooperation rather than unilateralism, and he announced a huge development initiative to bring Chinese models to the developing world.
The United States needs an answer. It should fund American A.I. companies to help bring medical advances, agricultural knowledge and other cutting-edge knowledge to the developing world.
The A.I. race is not like ones we have seen before. We risk giving away the world’s critical thinking to systems that promote the interests of a single government. This is a battle for the world’s mind, and the United States must win it. END COPIED MATERIAL
No mention that AI companies appropriate all the cultural assets to train their models for free, avoiding copyright by declaring "fair use", and even bypassing the robots.txt tag that is supposed to stop wen crawling everything. That, plus the lack of privacy laws in the US, allows companies to sell what was considered private information as part or all of their "business models".
The piece also suggests lightening regulations. That could mean forcing municipalities to allow datacenter locations, raising electric and water rates. It allows companies to power their datacenters with their own gas generators, increasing air pollution and climate heating. Noise is not trivial either. Given the general pushback against these facilities, and companies like Google forcing AI down our throats, whether we like it or not, there is a good argument to demand tighter regulations, not weaker ones.
My guess is that in a decade or two in the future, these will be obsolete white elephants, with newer types of chips drastically reducing the power and cost requirements of training and inference, and likely transferring the processing to local devices for much of the inference of "good enough" AI. And why should government be "picking winners" anyway? I thought this was not a good idea. Government should be funding research, not industrial development. We are likely to repeat the same mistake Japan's MITI made with its huge support of 5th Generation computing in the 1980s.
Hah, well-timed. I just Googled about this distinction, only to find you had just made a post about it. I just stumbled upon you on YouTube recently, and have become a big fan. And now I'm subscribed to your Substack! You're the man, Gary--appreciate the insight.
THAT is the difference. Whether you can make money from it or not.
Obviously, no one is going to release the training data, the exact training methodology, the bias, or anything else that would enable someone to just download and use it.
I run openSUSE Tumbleweed Linux on my machines (not necessarily the best choice, but there it is.) It's FREE. I can run it on a thousand machines, say on a VPS platform, if I want (and have the money for the VPS) for FREE.
OTOH, unless you are a Linux OS nut, you don't know how it works, either.
The difference is: if you ARE a Linux OS nut, you CAN learn how it works.
That's open source. The key word is SOURCE.
Which, for LLMs, means the training data, the training code, the weights, everything.
But almost all of the labs are financed by hundreds of billions of dollars by the hyperscalers.
The hyperscalers need to make money and the labs need to be able to pay back the hypersclers.
So NO lab is going to release open source.
The ONLY way you get an open source LLM is if the government funds a lab explicitly for that purpose from taxpayer funds and releases the results for free to anyone.
No one else is going to do that because LLMs are trained on massive amounts of data run through dozens, hundreds or thousands of INCREDIBLY EXPENSIVE GPUs and specialized chips.
Which cost MONEY.
So take your open weights and be satisfied.
Doesn't matter, since all the labs are going to go bust and we'll be left with just open weight models anyway. Or the frontier labs models will be sold off for cheap.
Or you can learn Python and make your own, slowly, laboriously, on your single GPU.
Asking for open source LLMs is like asking for an open source NASA Space Station.
"If you want to know how something works, you look at the code."
The problem with open source LLMs is that the code is the easy part to verify. However, the behaviour of the LLM is fundamentally based on the trained weights, which cannot be verified at all, however many people gaze at the values of billions of trillions of weights.
This has been the problem with all the pattern finding analytics algorithms since about 2002 and Big Data. It is almost trivial to check the code but the behaviour depends on the learned values and weights which cannot be validated at all.
I shoulda said that!!!
👇
https://x.com/i/status/2086863151002210631
the trained weights, which cannot be verified at all, however many people gaze at the values of billions of trillions of weights.“
Speak for yourself
Some of us can easily assess the value of AI’s just by looking at their weights.
That’s what scaling is for.
Like you can verify 2 trillion weights? Added to which all weights are between 0 and 1 in absolute values.
Easy
I just use my bathroom scale.
wtf are you talking about?
They can't expose their training sets without exposing themselves to fair use lawsuits. Speaking of which, how stupid is the US court system that judges think fair use includes taking every single bit of a company's data and using it to create a product that directly competes with that data for customers?
And how stupid are companies paying for the privilege of training the systems that will replicate their processes and replace their services (many, not all, of course)? Like all the manufacturers that outsourced to China. Either happily trading a few years of turbo profits or grudgingly being forced into it by price pressure and ending up dead after handing over IP and training the lower-cost competition.
Is this a reference to the Bartz v Anthropic settlement please? Or is there another case I should be looking at?
So far just that one, as far as I know, but the rest are coming. There is no universe where fair use can be construed to mean, "use every last drop of a company's content in order to create a product that will steal customers away from that content." That is the opposite of fair use. I'm astonished that there haven't been more cases yet other than Bartz.
Bartz is a fiction author who did not allege that any of their content ended up in the Claude models. The case was only about piracy of the books used for training the model to write like a human.
All is fair in love, war and AI
lol I do hope we're saner than to think AI is THE third exception to the rule of law, alongside such eternal edge cases as romance and literal war. AI is a product/service. It should be nailed to the ground like Gulliver among the Lilliputians. I'll be there with a nail gun to help.
And “OpenAI” is not the same as “open AI”
And “publicly available” is not the same as “public domain”
Turns out the reality distortion field in AI is pretty strong
And “annualized revenue” is not the same as “annual revenue” (aka “a null revenue” in the case of most AI companies)
I'll take your word that Zuck knows the difference between "open source" and "open weight". However, the first provided tweet on releasing the Muse code does not say it is either open source or open weight. The 2nd example, commenting on Nadella's tweet, could be interpreted as conflating the two, rather than "bait and switch." IOW, I read it as that it wasn't deliberate, but rather confusion.
i’d be astounded if he didn’t know the difference. his company deals with both all the time and he’s smart enough to get it.
Journalists would have zero credibility without the Gell Mann Amnesia effect. But y'all don't have to forget.
A good example of the phenomenon whereby I learn something new every day on Substack.
you had me at "for the love of Turing" <3
One could make a pretty good case that Turing is directly responsible for the LLMitation Game
no
Lovely to read your post on this, Gary~ This issue was itching at me and my colleagues a while back. So we wrote a paper on this topic not too long ago about how how the media discusses openness in AI (incl. how NYT have been using the terms) https://doi.org/10.1609/aies.v8i2.36688
TLDR, it's not just the NYTimes that get it wrong. Fun fact, there's no easy translation for "open source" in French and they use the English term.
And there's a lot more to the "openness" concept than just mis-labeling things as being open source (if you care more academic discussions, here's a taxonomy of openness we developed as well https://doi.org/10.1145/3715275.3732087).
I don't think any language translates 'open source'. Not Dutch, not German, not French.
Open weight is modern freeware. You can have it for free, can run it, and if you're a real nut, meddle with it with the LLM equivalent of a hex editor. The word "open" is smuggling a lot of unearned associations here, by design.
Thanks for the excellent summary. It leaves me wondering what these AI companies think people ARE supposed to do with their open-weight models. Perhaps they are hoping the community will invent new techniques for working with them. For example, perhaps we can train a model on common law only and then subtract it from one of their models, resulting in a new model that focuses on statutory law only. (Commenters, please don't tell me why this won't work. You'd be missing my point.) Are AI companies releasing these open-weight models solely in order to fool people into thinking they are actually open source?
Thanks for that diagram. I understood the distinction before (I specialized in FLOSS as a young lawyer) but hadn't seen it laid out like that.
Sadly this, basic, level of accuracy and understanding seems to be standard for modern mainstream media. And across all fields. It’s problematic to say the least.
This has been a problem for a long time in MSM. Science reporters with actual knowledge were replaced by those without it. These days, blogs by scientists (or other domain specialists) are often the best place to understand new discoveries or technology.
A personal case in point about expertise. On this blog or another, a comment I made was strongly corrected by someone with greater understanding of post-training than I. It proved a good "teachable moment" for me, which corrected my understanding, for which I am grateful. These expert blogs can be so much better than MSM reporting, and comments with dialogue can also be very instructive to add information, clarify points, and correct misconceptions.
Alex Tolley: See my "reply" that I moved to the general discussion.
Thank you. I replied to that very relevant post directly, although I took issue with the extracted section's arguments.
The distortions/misrepresentations are quite purposeful and the media are simply parroting them.
Even terms like “AI hallucination” are meant to make light of a serious, essentially intractable problem
no it isn't
Absolutely. And the adoption of words we apply to humans is completely intentional.
Most days I wish the whole field was called something else. The rot starts there and only gets worse.
For a detailed clarification framework for the many flavors of openness in AI, see this notable analysis: https://dl.acm.org/doi/10.1145/3778264
It's not just this one article. Many articles on AI from the NYTimes are like this where they report on something but don't do enough to understand the topic. By not understanding it well enough, they also aren't asking the critical questions to challenge claims being made by AI companies, for example, often taking what they say at face value or misinterpreting it.
Another thing I've noticed is that whenever they have an article criticizing AI, there is almost always a paragraph in there that says something like:
"Don't let this criticism detract from the amazing progress being made... AI is revolutionizing XYZ". Why can't a criticism just be a criticism without softening it?
Question for those more intelligent than me. I am struck by the prospect that Chinese models should be FAR more cost-competitive than American models, and I only see that disparity growing larger in the future. Because...
- They already have cheaper power than the US
- To my understanding, they are bringing an order of magnitude more power online in the next decade than the US (especially nuclear & solar)
- They can build stuff (i.e., data centers) far faster & cheaper than the US, and without NIMBY-ism concerns
- Subsidies from the CCP
This would seem to position companies like DeepSeek and Kimi very well, to be able to offer models that are, e.g., ~80-95% as powerful as US models, but at a fraction of the cost.
HOWEVER, I understand that their models are open weight, but NOT open source.
The question, then: fundamentally, to what extent can we really expect to see western businesses feel comfortable deploying these AI tools into their infrastructure in a meaningful way, absent the ability to truly audit the source code of these models coming from these Chinese firms? Or will distrust rule the day and prevent these Chinese models from ever taking off in a big way?
Dan: In considering China's motives, you might want to read the following article from the NYTimes, with clips:
The Hidden Cost of China’s Free A.I./July 29, 2026
https://www.nytimes.com/2026/07/29/opinion/ai-china-us-free-models.html?unlocked_article_code=1.1lA.pONz.K371DZnuiLmG&smid=url-share
Here is the last part of the above article:
ALL COPIED BELOW:
Still, labeling will not change the economics that make these models so attractive. Those economics are Beijing’s strategy. Chinese A.I. is both inexpensive and good. The United States still has the best A.I. models, but across many prominent benchmarks, the top Chinese A.I. models are only several months behind. Moonshot’s newest model, released in July, appears to closely trail the best Claude and ChatGPT models, although more assessments are still coming in. Even if it is well behind, it is free, like most leading Chinese A.I. models and unlike the leading American ones.
If Washington wants the world to use American A.I. models, it needs to match China on price and quality. The U.S. government should encourage American companies to cut prices on their older A.I. models that are as good as China’s. It should also support the development of safe, open alternatives to China’s models.
One would hope that when faced with two products of roughly the same price and quality, most businesses and consumers will choose the option that was not built to serve a foreign government. Some American companies may reduce prices on their own to prevent China from capturing the market. But they should not have to. Chinese A.I. is inexpensive because its government makes it so.
Washington should deploy its full arsenal to level the playing field, financing American A.I. infrastructure abroad and lowering costs at home by lightening regulatory burdens and building energy infrastructure. The government should keep exposing these models’ bias and should fund research into preventing Chinese A.I. companies from copying American innovations (primarily through a practice known as distillation).
Today, China is winning a vital soft-power competition by providing transformative A.I. to the world. President Xi Jinping gave a speech on July 17 at the World A.I. Conference in Shanghai that presented China as the noble and generous friend every nation would want. He described China as committed to open A.I., countering speculation among some experts that China might stop allowing the free downloading of its models. He also said his government was committed to international cooperation rather than unilateralism, and he announced a huge development initiative to bring Chinese models to the developing world.
The United States needs an answer. It should fund American A.I. companies to help bring medical advances, agricultural knowledge and other cutting-edge knowledge to the developing world.
The A.I. race is not like ones we have seen before. We risk giving away the world’s critical thinking to systems that promote the interests of a single government. This is a battle for the world’s mind, and the United States must win it. END COPIED MATERIAL
No mention that AI companies appropriate all the cultural assets to train their models for free, avoiding copyright by declaring "fair use", and even bypassing the robots.txt tag that is supposed to stop wen crawling everything. That, plus the lack of privacy laws in the US, allows companies to sell what was considered private information as part or all of their "business models".
The piece also suggests lightening regulations. That could mean forcing municipalities to allow datacenter locations, raising electric and water rates. It allows companies to power their datacenters with their own gas generators, increasing air pollution and climate heating. Noise is not trivial either. Given the general pushback against these facilities, and companies like Google forcing AI down our throats, whether we like it or not, there is a good argument to demand tighter regulations, not weaker ones.
My guess is that in a decade or two in the future, these will be obsolete white elephants, with newer types of chips drastically reducing the power and cost requirements of training and inference, and likely transferring the processing to local devices for much of the inference of "good enough" AI. And why should government be "picking winners" anyway? I thought this was not a good idea. Government should be funding research, not industrial development. We are likely to repeat the same mistake Japan's MITI made with its huge support of 5th Generation computing in the 1980s.
Hah, well-timed. I just Googled about this distinction, only to find you had just made a post about it. I just stumbled upon you on YouTube recently, and have become a big fan. And now I'm subscribed to your Substack! You're the man, Gary--appreciate the insight.
The bottom line in all this is, as usual, MONEY.
THAT is the difference. Whether you can make money from it or not.
Obviously, no one is going to release the training data, the exact training methodology, the bias, or anything else that would enable someone to just download and use it.
I run openSUSE Tumbleweed Linux on my machines (not necessarily the best choice, but there it is.) It's FREE. I can run it on a thousand machines, say on a VPS platform, if I want (and have the money for the VPS) for FREE.
OTOH, unless you are a Linux OS nut, you don't know how it works, either.
The difference is: if you ARE a Linux OS nut, you CAN learn how it works.
That's open source. The key word is SOURCE.
Which, for LLMs, means the training data, the training code, the weights, everything.
But almost all of the labs are financed by hundreds of billions of dollars by the hyperscalers.
The hyperscalers need to make money and the labs need to be able to pay back the hypersclers.
So NO lab is going to release open source.
The ONLY way you get an open source LLM is if the government funds a lab explicitly for that purpose from taxpayer funds and releases the results for free to anyone.
No one else is going to do that because LLMs are trained on massive amounts of data run through dozens, hundreds or thousands of INCREDIBLY EXPENSIVE GPUs and specialized chips.
Which cost MONEY.
So take your open weights and be satisfied.
Doesn't matter, since all the labs are going to go bust and we'll be left with just open weight models anyway. Or the frontier labs models will be sold off for cheap.
Or you can learn Python and make your own, slowly, laboriously, on your single GPU.
Asking for open source LLMs is like asking for an open source NASA Space Station.