Trump himself hasn’t entirely bolted — yet — but he’s certainly changed course sharply, moving from full anti AI regulation to strongly encouraging the voluntary preflight testing of frontier AI models.

Axios just reported this:

A new Super PAC called Guardrails Alliance, just put out a statement from their Political Director Charlie Blaettler which sums it up:

The Republican Party is panicking because they sold out to the AI industry – a while ago – and they know it. But now they are realizing how wildly unpopular Big AI’s agenda is with the American people. Now the GOP and AI’s unaccountable billionaires are scrambling to backtrack …

How they can backtrack is not entirely obvious. And of course to be fair some prominent Republicans like Hawley, Bannon, DeSantis and Blackburn have already made it clear that they are hardly fans of the AI industry in its current form. But the party is definitely in a pickle. And it will be interesting to see the degree to which Trump himself might further retrench, once he’s fully apprehended the writing that is now so clearly on the wall.

Share

Subscribe now

To begin with, hardly anyone trusts them anymore, which can’t be great for business.

Take the reaction to OpenAIs Sam Altman’s latest announcement, on Tuesday August 18 (i.e., “earlier today” for those of us on the West Coast), that OpenAI would be pausing, ostensibly for safety reasoning:

As far as I can tell, hardly anyone believed him.

The best one line translation came from the user @NIK on X:

Slightly elaborated:

And, here’s where things get worse. It turns out people were very likely right to be skeptical, because The Wall Street Journal’s Berber Jin and Corrie Dribusch just dropped big news:

and then an hour and half later, Jin dropped still further details, also bad for OpenAI:

You don’t want your quarterly losses to quadruple right before your IPO.

§

It’s safe to say all this is good news for Anthropic:

And terrible news for OpenAI.

§

Small wonder Nvidia was trying to cover its bets earlier this week:

Still, as Ed Zitron and I have both long noted, if OpenAI goes down the damage could spread:

In short, things could well get wild, very soon.

Share

Many people have written theories about what went wrong.

It clearly wasn’t money or a lack of resources.

Here’s my guess, from the outside, not about the obvious proximal triggers of what happened over the last few months (talent leaving, Gemini models falling behind, morale over military involvement) but about a more distal trigger that is maybe less obvious — a key decision made all the way back in 2023.

If my theory is right, the decision I have mind in might ultimately prove to be the costliest mistake that Alphabet’s generally excellent CEO, Sundar Pichai, ever made.

Read more

Share

Subscribe now

On Saturday, Dario Amodei claimed (in a long defense of his company’s approach to AI) that he thinks it “will actually be possible to cure most human disease in ~5-10 years”, highlighting “curing cancer” among other goals. As usual, some in the media ate it up.

And I wish it were true. But his timeline is so naive as to be absurd, both with respect to how medical science works and why it is challenging.

If you haven’t already done so, I strongly recommend you read (or reread) the March essay I wrote about AI and cancer, which builds on an essay by Emilia Javorsky. That essay started with Javorsky’s observation that (as of March) not one drug designed by AI had yet reached clinical adoption, even after over a decade of efforts.

Part of the crux of my essay (and hers) was that even if AI were a lot better, curing disease is hard, both economically and technically, and you also can’t simply skip the time-consuming clinical work. (I further argued that we don’t really have the right kind of causally- and biologically-sophisticated AI yet. New advances in AI and math don’t fundamentally change what I wrote there.)

The happy news is that you don’t have to take my word for any of this anymore. The mood has changed, and this time a whole pile of people pushed back. I have yet to see Amodei’s ridiculous claim dissected in the mainstream media, but below are some of the many tweets over the weekend that rightly took Amodei to task.

Let’s start with Eric Topol, a well-respected and very pro-AI physician/researcher who is having none of it. (Topol also notes that Hassabis made a similar absurd argument last year on 60 Minutes, maybe the most naive things I have ever heard Hassabis say. AI might someday contribute as Hassabis says, but certainly not in the next decade. Many of the quotes below happened to be directed to Amodei, who was in the news, but apply to what Hassabis claimed as well.)

By way of setting realistic expectations, Topol adds:

Caltech’s Bren Professor of Computational Learning, Lior Pachter, wrote

Computational biologist Keith Robison:

Cancer biologist John Herrman wrote (plausibly if optimistically, but nowhere near as optimistically as Amodei):

MD/PhD Marios Georgakis asks:

(As far as I know, the correct answer to the question about real plans posed in the first sentence is a simple “no”.)

The software engineer/enterpreneur Chombe Bupe, responding to Georgakis, adds:

Even the ultimate AI optimist, Wharton professor Ethan Mollick, wasn’t really buying:

Penultimate words to Dr. Catharine Young, a Senior Fellow at the Harvard Chan School of Public Health, in a brutal pair of tweets further underscoring Amodei’s naïveté with respect to medicine.

and

§

There are a great many things about Amodei that I respect, including both his technical chops and business sense, as well as his bravery in standing up to the US government on important matters. But his habit of giving people unrealistic hope is not one of them. (See also the time he claimed that AI could double human lifespan by the end of the decade.)

In fairness, Amodei may believe what he says, but in that case he ought take Dr. Young’s suggestion of doing a medical internship, in order to better calibrate his understanding of the challenges in medicine.

Share Marcus on AI

I am watching something fascinating unfolding in real time. Anthropic is in an SEC-monitored “quiet period” before its IPO (expected sometime in the fall), somewhat limiting its communications (though not entirely; Amodei discussed governance matters on X yesterday, a rare appearance there for him, and the company continues to report on safety issues and so on). But that hasn’t stopped leakers and generative AI bulls from trying to convince audiences that Anthropic’s finances are spectacular.

Aside from the Friday Reuters report above in which Anthropic is said to be “projecting 2028 ‌revenue of roughly $190 billion to $200 billion”, based of course on undisclosed math, “according to two [unnamed] people familiar with the company's financials”, we also heard yesterday about how amazing their Q2 was (further details below), by way of documents that were leaked to Bloomberg News. Anthropic may be “quiet”, but its fans are not.

Moreover, in a Friday All-In podcast interview, Gavin Baker claimed that Anthropic was making money on every token. A couple weeks earlier the prominent podcaster Dwarkesh Patel projected (in a claim he partly walked back, see below) that “Anthropic likely ends the year with ~$100-150B of revenue”

Below, I dissect these claims and explain what’s been left out.

Read more

And it is a sentiment that crosses party lines:

Anyone remember this?

Looks pretty on track.

Per The Information’s newsletter

The stock slipped a bit on the news [217.55 at close, though up a bit this morning in premarket], as this handy chart from FT reports.

For your convenience, I have highlighted the scariest bits.

§

Has Jensen Huang begun to overplay his hand?

Some people think so, like the financial journalist Holger Zschaepitz:

Last words in this brief essay go to Jim Chanos, famous for calling the Enron collapse:

Subscribe now

PS Some other breaking new to watch: Nvidia is releasing a new version of its truly open-source (not just open-weight) Nemotron model, which is good for end users and developers but that could also ultimately undercut places like OpenAI and Anthropic with whom it has deals.

It’s no exaggeration to say that open-source software — in which the complete source code is available to anyone — has changed the world and accelerated software development.

But open-source and open-weight are two very different things, and too many people —this morning it was editors and writers at The New York Times among others—are failing to distinguish the two.

The essence of open-source is twofold: transparency and customizability. If you want to know how something works, you look at the code. You can change literally anything about that code (“forking” a new version of that code, for example, if you want to make a different version). All of this is fantastic for the community, and that is why open-source has been such a dynamic force. It’s called open-source because you release the source code of the final product. Open-weight doesn’t do that.

Open-weight piggybacks onto all the great press that open-source has but offers few of the advantages. Open-weight models are NOT fully customizable, and not transparent. They are the product of machine learning (the weights of a trained neural network), but not the whole thing, from soup to nuts. That means you can run them (or “posttrain” them), but, crucially, users can’t customize or improve them to remotely the same degree (more about that in a second), and outsiders can’t investigate them to nearly the same degree. You also can’t build community in the same way.

Understanding all this requires a basic familiarity with the life cycle of a machine learning model. Let’s start with a simplified sketch I have snarfed (and modified) from geeksforgeeks.org. The highlighting, which I will explain in a moment, is mine.

(Not shown is “post training”, in which people try to align and bend trained “base” models to their will. That’s all you can do with an open-weight model; I will give you examples of what you can’t do in a moment.)

When someone releases an open-weight model they are releasing the (highlighted) candidate model [also sometimes known as a base model]—which is the output of a complex process—not the full pipeline that generated that model. You can adapt that candidate model, but you can’t rebuild it or investigate it to your liking, the way you could with a true open-source model, such as AllenAI’s Olmo or Nvidia’s Nemotron, which truly is what it says on the tin: “open weights, training data, and recipes.”

In contrast, when people release open-weight models, they are not revealing the Full Monty. They are not releasing the raw data; they are not revealing how they preprocessed that data; and they are not revealing the exact algorithms and parameters that they used in training. (For those who can stand yet one more loose metaphor, it’s like they are releasing a cake, but not the recipe; you can add icing on top, and maybe some fruit or hot fudge too, but you can’t change the raw ingredients that went into the cake itself.)

All this makes a world of difference, to developers who might use the systems, to regulators, to scientists, and ultimately to the world.

For developers, let’s say you have a hypothesis that you could make a safer AI if you delete all of the Reddit data from training. Sorry, out of luck: in an open-weight system (as opposed to a truly open-source system) you can’t change the training data at all, and in fact you can’t even see the training data. Or let’s say you would like to train a model on legal reasoning and recognize that common law and statutory law are different. To reduce confusion in the system you would like to filter out the common law stuff before training; again you are out of luck. Or suppose you would like to use Big Company’s training process purely on your own data, rather than on whatever random stuff they scraped from the web. Nope, can’t do that either. (Nor can you share the results of any of those experiments with the community, since you can’t run them in the first place.)

Or say you are a regulator, and you would like to know how much bias there is towards white males in the training data. Sorry, can’t look at that, either. What kinds of instructions are there relevant to bioweapons? Again, nope.

Are you a scientist? Want to investigate how much of what the model does is regurgitation and how much extends beyond the training data, which is arguably the central question in AI? Sorry, out of luck there, too. Those who released the open-weight model could do that, since they are privy to all these details that are actually closed, but you can’t. Or maybe you just want to know whether the answers to your favorite benchmarks are actually already in the training data? Nope, can’t look at that, either.

All of this is unfortunate. It means that developers often can’t really use the open-weight models (despite the hype) to develop what they want; scientists can’t use them to test a lot of key hypotheses, and regulators can’t peer inside the process.

Furthermore, if you wanted to know how much of the training data was copyrighted, you can’t look at that either — which is perhaps part of why none of the big companies want to release true open-source, and instead settle for open-weight, getting nearly all of the positive press without actually helping the community nearly as much.

§

What prompted me to finally write about this?

Meta just released an open-weight model. To his credit, Zuckerberg himself got the terminology right here:

But the Times did not:

Putting scares quotes around open-source does not make an open-weight model more open. And although one can post-train1 Meta’s new model, readapting it to some degree, for all the reasons I just described, it can’t be freely modified, nor used for proper science. You can’t do any of the modifications I described above. And you can’t figure out how the model got to be the way it is in the first place.

§

Zuckerberg knows perfectly well the difference between open-weight and open-source. A few weeks ago he was extolling the value of open-source:

But also played bait and switch with another tweet here:

And then today he released a model that is NOT open-source, hoping you wouldn’t notice the difference between the two. And, sadly, the Times didn’t.

I implore them, and everyone else, for the love of Turing, please don’t confuse open-weight with open-source.

Subscribe now

P.S. Bonus analogy; pick your favorite:

Classic computation mostly uses CPUs, which are more general-purpose in the kinds of computations that they are designed to perform.

Neurosymbolic AI, which by definition tries to combine the two, typically requires both.1

In the age in which pure neural networks were dominant, which one might very roughly date as 2012 (when people began running neural networks on GPUs rather than CPUs) through mid 2023 (when Frontier companies quietly started incorporating things like symbolic code interpreters), a very large fraction of commercial AI was driven almost purely by GPUs.

Now we are seeing something different.

[This is a paid post; I have not done them often but recently read that the Substack algorithm prioritizes them, and am curious to see what happens. It is also a gentle reminder that I do appreciate your support.]

Read more

In huge news, in case you missed it, Sir Demis Hassabis has given up the reins at GoogleDeepMind (moving aside to be Chairman and Alphabet’s Chief Science Officer), and the rightly legendary Jeff Dean and some other top talent just left altogether, following earlier departures of star researchers like John Jumper, David Silver, Tim Rocktäschel, and Noam Shazeer. Google/Alphabet are hemorrhaging talent, morale is poor, and their latest model is delayed. Many are ready to count them out altogether.

I think this is a mistake. Here are seven reasons why:

1. Google still has access to an enormous amount of data, from search and email especially, at a scale that nobody else has.

2. Only a small number of companies (Meta, Microsoft, Amazon, maybe a few in China) can match on sheer compute and expertise building distributed systems. Google/Alphabet has long had their own TPU chips, lessening their dependency on Nvidia.

3. Google still makes a ton of money, $402 billion in revenue last year, $132 billion in profit), and has vastly more cash to play with than OpenAI and Anthropic. (The fact that Google is going to the bond market yet again is worrisome, though.)

4. Google has great distribution, through Android, Mail, Search, Docs, YouTube etc.

5. Hassabis, arguably the most formidable person in AI, is still there, not gone altogether. He has worked together with his replacement, Koray Kavukcuoglu, for over a decade, and they will continue to work together. (I have spoken once with Koray and found him to be smart and sober, in a world where not all of their competitors are. His lack of name recognition may be a liability in the short-term, but his sobriety may be crucial in the long-run.)

6. Google’s most obvious competitors have troubles of their own. OpenAI has a declining reputation, Anthropic has conflict with the administration, Microsoft and Amazon don’t have the same talent pool.

7. Google can probably keep going even if it doesn’t take much of the LLM market in the end (which I expect will be a multi-way tie with relatively modest profits). On the other hand, if things do turn out as I expect, and there is a multi-way tie with relatively modest profits, Anthropic and OpenAI don’t have that much else to fall back on; they may never earn out their investments. The only real existential threat would be if Google loses Search, Android, and YouTube altogether. Google may simply outlast the other two.

Google definitely doesn’t look unassailable like they once did, and may never be as strong as they once were, but it’s premature to count them out.

Subscribe now

The idea that robots will be the best surgeons on earth anytime soon is absurd, and I challenged him about it on X:

Why did I bother to offer such a bet? Here are three reasons.

• Elon’s timeline predictions are notoriously, and to my mind, obviously, bad, yet all too often reported as gospel. I have used bets as a way to get people to take them less seriously. The last time I offered him a million dollar bet (offer raised to $10M by Damion Hankejh) was back in April 2024, after Musk claimed that AI would exceed the intelligence of all people by the end of 2025. That of course didn’t come to pass. Indeed, in his recent interview with The Economist Musk himself quietly backpedaled to 2031, shades of his regular re-predictions around autonomous vehicles. The surgery prediction, too, will surely be repackaged with a new date, at a later date. Elon Musk is, like a lot of LLMs, frequently wrong and never in doubt. Enough already.

• Regarding robotic surgery timelines, Musk is so deeply wrong it’s hard to fathom. I asked esteemed roboticist Rodney Brooks what he thought of Elon Musk’s claim, and he fired back with a firm opinion five minutes later, describing Musk’s claim as “Completely crazy talk”, elaborating as follows:

  There are no lab demonstration remotely close to doing surgery on a living test animal, even with a set of arms solidly affixed to the ground rather than being on top of a two legged balancing platform. Just the problem of seeing what is happening in the way a surgeon does is not even lab demonstrated. What we will see over time is more non-invasive surgeries and they will be a team of AI and geometry based sensing along with expert surgeons over the next few years, and only then will the stuff be good enough to use on pigs, and only many more years of refinement will we get to FDA (or whichever agency) that lets these things be tried on humans. See my weekend blog post at rodneybrooks.com/blog on how everyone gets time scales wrong.

  When I asked Brooks about the state of the art in existing robotic surgery trials, he added this

  All of them still have a human in the loop. Mako surgical from Stryker (I was the first VC who invested in them around 2000) for knee replacement, is closest, as it is not soft tissue but bones that can be accurately imaged with xrays during the surgery, and it is largely geometry, but then a whole lot of stuff the human worries about and supervises so that it doesn’t accidentally damage blood vessels or nerves.

  There is just no way all this gets sorted out by the end of the decade. What Musk told Diamandis is just not happening.

• Just as Hinton’s famous (and famously wrong) 2016 pronouncement about no longer needing to train radiologists probably scared away potential radiologists, Musk risks scaring away potential surgeons. That’s really bad, especially that his premise is probably off by a least a decade in the best case and quite possibly more.

It would be nice if these Tech CEOs were more accountable for what they said. That’s the real reason I offer these bets. And the real reason the CEOs never take them.

P.S. Jason Calacanis of the All-In podcast has expressed some interest in turning this debate into a bet on Polymarket. I will report back if he commits. For criteria I pointed a list to ten difficult surgeries as a possible metric; no word back thus far.

My jaw dropped a couple hours after my own post, reading this from Levent Alpöge, an accomplished mathematician who works at Anthropic:

One (obviously smart) guy at Anthropic was able to “replicate” (as best he could given OpenAI’s very sparse report of what they actually did) half of OpenAI’s results within 24 hours. And not with Astra or some other new model, but with the already publicly-released Fable.

I had a couple reactions to this.

a. Alpöge’s fast partial replication made me wonder whether OpenAI’s real advance here was finding (through AI) which open problems were amenable to a certain kind of search and verify technique. OpenAI’s Noam Brown acknowledged, without much detail, that there were failures on some (and maybe many) others:

Unfortunately, beyond that OpenAI does not report which problems they tried and failed at. They have given us a numerator without a denominator, always a worrisome sign.

This doesn’t take away from OpenAI’s accomplishment - if the system solved a bunch of open problems that’s very impressive – but it again raises questions about what the scope is: all math? A certain subset of problems that can be searched in a certain way?

Maybe once that subset has been flagged, maybe many systems can do them. That’s kind of what Levent Alpöge‘s partial replication shows.

And many what was most critical was not some innovation in Astra per se but the idea (probably from a human) to do this particular search.

b. Alpöge‘s fast partial replication undermines the widespread presumption that Astra is a major breakthrough. Astra is an obviously an advance to some degree, but that degree may well turn out to be merely incremental relative to other recent models.

Related to this, I was interested to read in The Information this morning that OpenAI hasn’t decided whether to call Astra (which was an internal code name) GPT 6 or GPT 5.7. If OpenAI truly thought Astra was a quantum leap,, I doubt that would be such a hard choice.

How good Astra is in open-ended real-world problems very much remains to be seen.

§

Second, a longtime reader of this newletter, Ihor Gowda, pointed me to a fantastic lecture given by Terence Tao, by any measure one of the world’s leading mathematicians; many see Tao as the leading mathematician. His lecture is lucid, accessible, and genuinely important.

What’s more you don’t need to be a professional mathematician to read it. It should be mandatory reading for anyone thinking about how AI will affect mathematics, and strongly recommended for anyone who cares about AI’s impact on the world.

One of the issues that he raises he calls “proof indigestion”; what happens if AI creates a lot of stuff, much of it true, but not necessarily useful:

Tao’s July 26th lecture, written before the Astra announcement, also prefigures another key point that I made yesterday: being good at creating certain sorts of proofs doesn’t mean that an AI system can contribute to all (or even most) of the goals of mathematics. He too distinguishes between solving open problems (which Astra seems strong at, perhaps with some important limits) and building theory (no evidence thus far that Astra can do that):

Tao is no Luddite, or even anti-AI. He is very open to AI, and eager to find a place for it in mathematics. His lecture is a beautiful read on how to do just that — to bring AI into mathematics, as thoughtfully as possible. Highly recommended.

P.S. Cartoon of the day, illustrating Brandolini’s law:

Noawadays, as the engineer Wouter Vreugdenhil, who brought the cartoon to my attention, points out, “that’s the game.”

Part I: The Biggest Fallacy

But at the same time, a whole raft of people, some fairly prominent, are running around making a deeply flawed argument about the implications.

See if you can spot the fallacy. Here are three examples among many.

Another tweet went so far as to claim that “the species just crossed a one-way threshold”; Elon Musk took it as evidence that we had reached The Singularity.

Each is making essentially the same error.

§

The fallacy has a name; it’s called the fallacy of composition.

The wiki on it is filled with great examples; here are just the first few.

What’s the manifestation of the fallacy in the current case? Thinking that a system that is great at a certain kind of math problem is great at all math, great at science or even quite possibly great at everything. The “AGI-is-near” community keeps committing the same logical fallacy over and over. Every time there’s an advance, I see the same error.

Here’s how the fallacy works.

1. Someone pretends that all cognition is created equally. (Totally untrue.)

2. Whenever AI achieves success on some form of (fancy) cognition, they want you to believe that success on all forms of AI is imminent.

You don’t have to be a cognitive psychologist to realize that this inference just doesn’t follow. We all know, for example, that expertise in math doesn’t guarantee genius in all domains. Someone who is great at math or physics or programming may struggle with writing or understanding human relationships (and conversely a great writer may be weak at math, etc). Expertise in one domain does not at all guarantee expertise in all or even most domains.

That’s precisely *why* people like Howard Gardner and Robert Sternberg developed multidimensional theories of intelligence, why the SAT tests math separately from verbal, etc.

Astra appears to be —we still haven’t seen the methodology—great at math, or at least some forms of math, but that does not mean that it will avoid hallucinations or solve the reliability problems other GenAI systems have. It doesn’t even mean it will be able to read PDFs reliably. And it doesn’t mean it will be the first generative AI to be able to obey hard rules, either. (Which should terrify you.)

In particular, Astra is obviously excellent at some problems; but that doesn’t mean it will be excellent or even competent at problems that are hard to formalize. It doesn’t mean it will be magic. It doesn’t mean it’s AGI or ASI or any of that.

It’s *very* impressive. But I see no reason whatsoever to think Astra is AGI let alone ASI. If it can score even a 5/10 on my 2024 bet with Miles Brundage I will be surprised.

Part II. There is an important, principled reason to think that success on math is a special case which will not generalize as much people might hope.

I wouldn’t go on about this fallacy at such length if (a) it wasn’t wildly common and (b) I didn’t think that there was very good reason to think that the fallacy applied here.

It’s not an accident that math is where these models are shining. Math lends itself to two things: verification (using symbolic tools), and massive amounts of cheaply produced synthetic data where you can guarantee that the answers are correct. The same applies to coding — but it is not true in general. You can generate as many math facts as you want; you can’t simulate the open-ended world. You can verify math; you can’t verify a military strategy in the same way.

This is not news. I have been pointing this out at least since January 2025 [first line should have said coding and math], echoing something Ernie Davis and I said about Go (and why AlphaGo would not be a panacea) in 2019:

OpenAI can do what Astra does in math because math allows for external tools to do verification and to create synthetic data.

That doesn’t make Astra less impressive, but as we learned a decade ago from IBM’s shambolic and ultimately failed attempt to turn Jeopardy-winning Watson into a cancer-fighting machine, success in one domain does not guarantee success in all.

The kind of math OpenAI is dealing with here is radically different from many (perhaps most) real-world problems in which verification neither guarantees solutions nor allows one to produce infinite training data effectively for free.

To expect it to be a universal solvent is to show you don’t understand that basic fact.

Part III: Seven more things to know about Astra

1. Yesterday's tweet and blog were marketing, not science. Neither of those nor the 249-page math article that went with them give any information about how this was accomplished:

   In an email commenting on the first draft of this essay Ernie Davis added, quite rightly:

   To properly evaluate the significance of these discoveries we would need some
   critical pieces of information.
   
   First: How many conjectures were attempted? If the OpenAI team picked these 10 conjectures at random from the space of all outstanding significant mathematical conjectures and Astra solved all 10, that would be amazing; but that seems altogether unlikely. If they cherry-picked 50 conjectures that they thought Astra would succeed on and it succeeded on 10, that’s still amazing, but significantly less so. If they ran Astra on all 1000 or so open Erdos conjectures and on 10,000 other open conjectures, then its failure to solve those is significant information on its limits as a mathematical reasoner.
   
   Second: OpenAI brags loudly that this was done for a total computation cost of $2000. It seems a safe bet that this includes only the conjectures where Astra succeeded, not the ones where it failed. But what was the cost in terms of the salaries of the highly-paid mathematicians and computer scientists who worked on this project? I’d be astonished if it was less than $20,000 and would not be surprised if it was upward of $200,000.

   We don’t know this, and the way things are going, we may never know this. We still have zero information about the OpenAI system that achieved gold-medal performance at the 2025 International Mathematical Olympiad, and very little information about the Google system.

2. Astra is good at some math but it is very doubtful that math is “solved” (as a lot of people on X seemed to believe). It seems to be good at certain kinds of math

   

   But quite possibly not all. Here are some examples from Eric Weinstein on X about some kinds of math it might be less good at

   

   Ernie Davis also adds “The problem of autoformalization --- turning mathematics written by a human mathematician into a strictly logical form -- does not seem close to being solved, though certainly progress has been made. For example: For the last two years, mathematician Kevin Buzzard has been leading a project to formalize Andrew Wiles' proof of Fermat's Last Theorem in Lean. It is quite safe to say that no AI system is capable of carrying out that monumental task. Presumably if they could do it unassisted, they would scoop him, and if they could cut down his work from years to weeks, he would be using them.”

3. Consistent with my general prediction that facility in math doesn’t guarantee universality, it appears that Astra’s proofwriting is not on par with the proofs themselves (echoing something Ernie Davis and I observed here a year ago)” [click through for full tweet]

4. I seriously doubt that Astra will magically solve all the things that don’t work reliably in current models. Will it be able to reliably extract numbers from arbitrary PDFs? Doubt it. Will it solve Sabine Hossenfelder’s desire to have AI write scripts [click the tweet for more details] for her YouTube series? Doubt it.

5. Astra is not (as Musk suggested) the Singularity. Enough said. (But see my recent essay on the topic; nothing has really changed).

6. It is absurd to claim that the Astra debut is “plausibly the most significant day in the history of mathematics” as a bunch of people went around saying, based on a quote from one run of Fable (which might give different answers if asked twice). There was no new theory, no new techniques, and as impressive as it is, it is not on remotely on par with the development of calculus, algebra, logarithms, probability, the decimal system, information theory, the concept of zero. Says Davis “The claim quoted from Fable that "several of those would individually have been the result of the decade" is absurd … As a reality check, 14 of David Hilbert's 23 problems have been solved since 1900 --- that's one every nine years --- and these results are nowhere near that league. You could easily compile a list of 100 much more important results that have been proved since 1926.”

7. I doubt it will lead directly to a cure for cancer, or enormous advances in “materials research, energy production, drug discovery, Everything”, as some excitable AI “influencers” envision. Nor does this mean we just entered “the era of automated scientific discovery”. Against overwrought claims like these…

.. this just came out, echoing the new article by and others that I linked to a few days ago.

Part IV: Summary

Despite immense pushback on X yesterday (most of it ad hominem, some literally involving fabrication, a lot of it involving outright lies), my overall hot take on Astra remains what I first posted:

What’s sad about this is that Ernie Davis and I raised a lot of the same points about how one could properly examine new systems a year ago:

I get that people are excited about Astra, but anyone hoping for magic is likely to be disappointed.

Subscribe now

You can read the full report here.

Now for the reactions (the middle one is mine).

§

The careful attention to language and social responsibility award goes to the prominent and increasingly outspoken investor Bill Gurley:

Old days: “Mistakes were made”; new days “Claude did illegal things”.

§

A journalist just asked me, “I saw you mentioned Anthropic’s AI oopsie and want to know what you think about this and the OpenAI hacks as a big picture on AI and cyber?”

I think it’s bad:

The supposed leaders in AI and AI safety are clearly in over their heads. Worse, we as a society are in over our heads. From a technical perspective, this is what happens when you let pattern0matching machines with no real comprehension of what they are doing freely roam the internet. It’s everything that I have been warning about for years. From a societal perspective, it’s insane that we are pouring gasoline onto the fire by racing ahead with data centers and AI rollout when we have no idea how to control it and are just sort of hoping for the best. Unfortunately, as the saying goes, hope is not a strategy.

There’s an important footnote on the “freely roaming” the internet part, by the way. What really allowed this to happen was human error at Anthropic, in allowing the pattern-matchers to roam the internet in first place, as key passage in the report notes:

If we can’t even trust Anthropic to doublecheck their sandboxes…

§

And, finally, prize for best skewering goes to Joann Stern, formerly of the WSJ and now launching her own NewThings.com.

Subscribe now

P.S. Some words of advice to the AI companies from Zack Korman of Embroidery.io:

No time today to cover all this in depth, and I might have missed one, but make sure to check out:

1. The dramatic, hubris-fueled, hyper-leveraged downfall of Leopold Aschenbrenner’s high-flying hedge fund Situational Awareness. You can read the NYT coverage here, or this short summary that appeared on X:

   

   .

2. The U.S. government shared a map of Africa that mislabeled every single country, spotted by :

   

3. Bonus: Reuters further reports that “the map included in the presentation contained an artificial intelligence watermark that ⁠signals it was made with OpenAI tools. The company said it was investigating the report.”

4. OpenAI was driven by competition to drop its prices by 80%. Who needs margins anyway?

5. Anthropic? They had a little cybersecurity oopsie of their own

   

6. MIT Technology Review reported new, disturbing adventures in jailbreaking

   

   :

7. and a large team of others reported that AI scientists still aren’t really up to scratch, especially when it comes to open-ended research:

   

   Who knows what tomorrow will bring.

   Subscribe now

A lot of people have come around to that view:

You can read the full story—which reviews a wide range of concerns, from business practices to a sentiment that the doomerism has been overdone to concerns about whether Anthropic is trying to kneecap competitors—here. For many, the turning point seems to be the events of the last few days, when much of the industry but not Anthropic came out vigorously for “open-weight” models (which can be downloaded, given sufficient hardware, but which are not fully open source, because e.g. their training sets are generally not publicly available). Many signed a letter; Amodei instead issued a statement explaining his resistance, which many saw as tone-deaf and self-serving, in part because the statement called for locking down distillation by competitors even as it just came out that the company has been destroying rare books in bulk in order to distill their contents.

I found myself (somewhat unusually) heartily agreeing with David Sacks:

All told, it would appear that most of the goodwill that Amodei accumulated in standing up to Hegseth has dissipated, much as most of the goodwill towards Altman eventually did.

Maybe one or the other, Altman or Amodei, will regain the public’s trust, but it is far from obvious that they will. In Altman’s case, I think dishonesty and lack of candor is the core issue; with good reason, people just don’t trust him anymore. For Amodei, who maybe means well, I think the issue is more one of tone deafness and self-certainty, coupled with a tendency to present arguments that frequently appear self-serving.

With apologies for the AI slop (the text at the bottom is particularly hilarious), I made a meme in their honor. Coming soon: a guy who could sell refrigerators to Eskimos (probably lying about their future benefits) versus a guy who often seems unable to read a room:

More seriously, the same WSJ article noted above reports that “Technologists and policymakers say they are also concerned about the AI race turning into a duopoly, with Anthropic and OpenAI essentially emerging as dominant players that control the market.”

Given how damaging AI could ultimately be, we need people we can fully trust, and I don’t think we have that.

Subscribe now

P.S. I started a poll on X an hour ago. It’s not remotely scientific of course, but the early returns are pretty clear. The good news (for Dario) is that they prefer him to Sam 2:1; the bad news is the vast majority of respondents don’t trust either of them.

Am I the only one who thinks that spending a few trillion dollars to prop up a couple companies run by people hardly anyone trusts is bonkers?

Remember “CEO said a thing” journalism?

A lot of CEOs just said a thing:

Don’t take them seriously.1 Partly because, well CEO’s gotta CEO2: In the modern era, making outlandish claims is practically in the job description:

I don’t even know why Sam’s singularity remark (made recently during a podcast) is even news, since Elon has been saying it all along, e.g., in January

and he more or less said it last year, too.

and as far back as 2017, longer than he has been touting fleets of Robotaxis:

Sam, as is often the case, is simply aping Elon. (And for that matter Sam himself said it a year ago, in a self-contradictory blog that never defines the term.)

§

Another reason not to take the whole lot of them seriously is that, so far as I know, none of them has actually defined what they mean. So far as I know nobody other than me even bothered to ask. As Neil Turkewitz put it (replying to me):

§

No matter how you slice it, we just are not actually there yet.

If we were, we wouldn’t be once again seeing what I have been calling the Klarna Effect, the AI hype two-step, in which employers boast of using AI to reduce head count only to quietly rehire humans.

But here we are:

According to the WSJ, “The push to expand head count, at least modestly, is a reversal from the prevailing corporate messaging during much of the AI era. Major employers largely held back on adding people due to economic uncertainties or a belief that artificial intelligence could shoulder more tasks on the job. But some executives say the costs and limitations of AI now demand that more people be added; others want to hire people back following layoffs.”

§

And we certainly haven’t reached the age of abundance that Peter Diamandis and Elon Musk keep insisting is a natural consequence of AI.

§

Nor have we reached a moment in “which technological growth accelerates beyond human control”

We can still perfectly well shut this stuff off if we want to. (It’s mainly because of greed that we don’t, stepping on the gas rather than the brake.)

That much-discussed OpenAI hack of HuggingFace? A human at OpenAI launched it; with a physical firewall it wouldn’t have happened. It wasn’t spontaneous combustion; it was a drill.

Had OpenAI ordinary guardrail classifiers been in place, it wouldn’t have happened. It doesn’t count as “uncontrollable” if you have to give the system a head and shoulders head start. (Yes, we do have poorly controlled AI, but that’s not new, either.)

§

Another definition of the singularity, per a TIME interview with Ray Kurzweil, who is often associated with the term, is “a moment when humans merge with AI via brain-to-computer interfaces to become “much smarter””.

We are clearly not there yet, either.

§

Another notion is that a singularity is a moment in which we can no longer peer forward into the future, but I don’t think we are there yet, either, as an entrepreneur on X pointed out, rightly ripping intp all the recent Singularity talk as nothing more than PR:

§

Going back to a justly famous paper by I. F. Good in 1965, the crux of the Singularity is supposed to be something far beyond any human:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

— Speculations Concerning the First Ultraintelligent Machine (1965), IF Good.

In this sense, the singularity is usually supposed to mean a step beyond AGI that can do anything a person, even an expert, can do, and much more.

The idea that we have reached that point is laughable. Here are 10 examples of things that “as good or better than the best human” AGI ought to be able to do, borrowed from the bet I made here 18 months ago with Miles Brundage:

The number of the above that actual current AI can do is still by my count no more than two.

And one could easily come up with many more. Which is part of why a bunch of companies are quietly rehiring.

§

None of this means we won’t someday (most likely not this decade) reach a moment of superintelligence. And if we don’t radically up our game on AI safety, it could be out of our control. Maybe it will help lower prices; maybe it won’t. Brain-computer interfaces will surely improve greatly (though it may take many decades or even centuries to get where Kurzweil dreams of).

But intelligence is multidimensional; we may have made lots of progress on some, and not so much on others. I doubt there will be a single magic moment that our descendants think of as the “Singularity”. If there is, it’s not one that’s going to happen in this decade.

CEOs can say whatever they like. That doesn’t make them true.

Subscribe now

P.S. The good news is that you don’t have to be a CEO to make stuff up. Here’s a projection from May that got 1.4 million views that now, two Chinese models and the death of tokenmaxxing later, looks like it has very little chance to be true:

The market went wild. Oracle stock briefly went up by 43% (and ultimately even a bit higher). Larry Ellison’s net worth briefly leapt by $100 billion. The Wall St. Journal gushed:

The whole thing never made sense though. The next morning, inn an essay called Peak Bubble I warned that

Yet Oracle continued to rise!

But not for long. Its stock stood at $307 per share at the close of trading on September 10, and briefly closed as high as $328 ten days later. Now, as I write this, it’s hovering around $120.

In the intervening nine months, the market has grown wary of all this circular financing stuff. Mocking illustrations like this one (from yesterday) have become widespread:

§

All of which brings me to the present. Yesterday, news broke that Nvidia is considering offering a $250 billion backstop for an OpenAI-led data center. Which led last night to considerable skepticism

and this morning led to a dramatically different reaction from the market, nothing like the bounce like Oracle had back in September. Instead, in the first couple hours of trading Nvidia is down over 4.5%:

The market isn’t reading the latest potential Nvidia-OpenAI deal as good news; they are reading it as desperation. As one investor put it to me in an email this morning, “[the deal] seems incredibly desperate by both parties. Two drunks leaning on each other to stand up.”

§

Meanwhile, Apple, the butt of jokes for a while because “it didn’t invest in enough AI” just overtook Nvidia

and SpaceX, which tried to position itself as an AI company is in free fall, dropping over 3% today, and by over 25% in the last month, more than 50% from its June high of 225:

§

Meanwhile, creditors are getting pretty leery of the whole thing, too, as Axios just reported:

And then there’s all the off-balance-sheet financing that is coming to light. And the sudden need for so-called “creative financing”. If that’s not a warning sign, I don’t know what is.

The GenAI boom hasn’t been sustained by profits; it has been sustained by hope and circular financing. That may no longer be enough.

Subscribe now

Update: just made an important additional point I should have made:

This posted in response to a tweet by Commerce Secretary Howard Lutnick

The CAISI report is here.

While I don’t doubt the CAISI report per se, I am troubled by Sacks’ take, and wrote about it here:

As argued here a couple days ago, I think the US should at least consider pivoting and trying to work with China towards AI for global good, rather than creating a new cold war that is likely to an end in uncomfortable way for all, with no clear winners.

Subscribe now