BREAKING: Sam Altman concedes that we need major breakthroughs beyond mere scaling to get to AGI

It’s past time to look for new architectures

Another dramatic sign of changing times: Sam Altman, who ridiculed my 2022 critique of LLMs that argued that scaling would not bring us to AGI and that we would need architectures has just argued that

… on the research perspective, I bet there is another new architecture to find that is going to be as big of a gain as transformers [were] to LSTMs … So I would go look for where I can find a mega breakthrough [with AI’s help] …

You can watch here.

Note that in this talk Altman didn’t claim to have found such an architecture. That represents a significant retrenchment from his claim fourteen months ago that “We now know how to build AGI as it’s usually understood.”

§

Taken together with Musk’s recent admission that xAI was “not built right” and Zuckerberg’s delay of Meta’s latest model, the shift from three prominent tech CEOs in a short period of time is a strong sign that insiders are losing faith in pure scaling.

Hassabis is no longer on board either, and nor are Sutskever and LeCun. Nadella and Pichai have also hinted at skepticism around scaling.

The view of this substack since its inception has been that scaling would not lead to AGI — and it hasn’t.

§

Yet bafflingly, the powers that be are still contemplating spending trillions on data centers that are costly to our environment and that might ultimately require government bailouts.

With the case for scaling as a road to AGI steadily crumbling, it is time to reconsider. The bad bargain of AI data centers makes no sense.

Image created by Di Rifai.