Influencers have gone wild lately claiming that AGI is imminent. One even alleged last night that “Sam [has] just called an emergency meeting.. . we’re way ahead of schedule. weeks not years.”
Sam Altman himself hinted we were close, just two weeks ago, writing “We are now confident we know how to build AGI as we have traditionally understood it”, which to be honest gave me Elizabeth Holmes vibes.
Today he walked it back.
Gary Marcus stands by his prediction that we will not see AGI in 2025, and regrets that Elon Musk didn’t take his million dollar bet.
I advise caution against buying into the hype created by OpenAI. They have repeatedly made claims that have later been discredited by independent research.
There are also significant issues with AI benchmarks, particularly given that the training data is neither shared nor independently verifiable, leaving them open to manipulation.
This appears to be yet another case of what resembles p-hacking – manipulating data to produce results that align with one’s interests.
Consider this excerpt from a tweet:
“Remember o3’s 25% performance on the FrontierMath benchmark?
It turns out OpenAI funded FrontierMath and had access to a substantial portion of the dataset.
The mathematicians who created the problems and solutions for the benchmark were not informed that OpenAI was behind the funding or had access to the data.
This raises critical concerns:
• It is unclear whether OpenAI trained o3 on the benchmark, making the reported results questionable.
• Mathematicians, some of whom are sceptical of OpenAI and would not wish to aid general AI advancements due to concerns about existential risks, were misled. Many were unaware that a major AI company funded the effort.
From Epoch AI:
‘Our contract specifically prevented us from disclosing information about the funding source and the fact that OpenAI has data access to much but not all of the dataset.’
https://x.com/mihonarium/status/1880944026603376865?s=46&t=oOBUJrzyp7su26EMi3D4XQ
Sam Altman: “guys please stop believing the hype I’ve played a huge hand in creating,” 🙄. I feel like Sam Altman’s corporate title should be “Chief of AI Hype,” because that seems to be his whole job.
Also, Gary you’re the best.