Intelligence (which I equate with problem-solving) has 3 scalable dimensions: (1) "inventiveness" (basically, how good the underlying problem-solving mechanism ("inventor") is at solving any problem X given information Y), (2) the knowledge/information Y that guides the inventor towards solutions, and (3) the physical resources (including time, energy, compute etc) that are/may be consumed by/during the problem-solving process. Given that they are engaged in a race to AGI, the major AI labs always go for the low-hanging fruit, such as (a) scraping data from the internet, (b) synthesising new data, (c) throwing $$$ at compute, and (now) extending compute time, i.e. (2), (2), (3), and (3). Anything, of course, but (1), i.e. the *actual* problem. Because (1) is hard!
Microsoft’s AI digital assistant, Copilot, has raised concerns among Microsoft's employees and executives about its ability to deliver on its ambitions of being the product that will “fundamentally transform our relationship with technology.” According to an October 2024 survey by Gartner, just four out of 123 IT leaders said Copilot provided significant value to their companies.
All the scaling has failed since general intelligence is just that. Factual statements by LLM are missing the human test. They must get the facts right. The missing component is explicit knowledge. Only semantic AI models (SAMs) have the ability to certify results. Here is a way that a SAM could improve training while markedly cutting costs.
The problem is that nothing actually scales permanently, getting better on a linear trajectory.
Every phenomenon follows an S-Curve and plateaus eventually.
In the case of LLMs, we could see this with the data that was published in the 2022 "Emergence" paper when the graphs were replotted from the deliberately misleading false origin and log X-Axis to standard true zero-origin and standard linear scale the plateau is very clear.
Inference time and/or efficiency probably is the next scaling frontier. These models have the entire corpus of human knowledge. So the answer is in there somewhere. You just have to figure out how to pull that needle out of the proverbial haystack.
I just came across a video of a talk by Thomas Friedman is which he did a very hard sell of the idea that we will have AGI in the next few years and that it will "transform everything". I really don't understand where this is coming from. It seems to be a kind of naive (or perhaps sinister) stand in for the likely outcome: that (possibly very) useful AI will make many people more productive and that "everything" will be "transformed" only because those with power will demand that everyone do more, faster, and better — or be replaced. In other words, it sounds to me like it's a way of framing what is in fact a likely power grab by capital and elites as a kind of technological inevitability that we will just have to come to terms with, rather than something that is fundamentally political and social, that we (ought to) have the power to resist or shape to our benefit.
2024
Me: Alexa, tell me about information science
Alexa: here’s some misinformation and pseudoscience.
Intelligence (which I equate with problem-solving) has 3 scalable dimensions: (1) "inventiveness" (basically, how good the underlying problem-solving mechanism ("inventor") is at solving any problem X given information Y), (2) the knowledge/information Y that guides the inventor towards solutions, and (3) the physical resources (including time, energy, compute etc) that are/may be consumed by/during the problem-solving process. Given that they are engaged in a race to AGI, the major AI labs always go for the low-hanging fruit, such as (a) scraping data from the internet, (b) synthesising new data, (c) throwing $$$ at compute, and (now) extending compute time, i.e. (2), (2), (3), and (3). Anything, of course, but (1), i.e. the *actual* problem. Because (1) is hard!
Gary have you seen this study?
Microsoft’s AI digital assistant, Copilot, has raised concerns among Microsoft's employees and executives about its ability to deliver on its ambitions of being the product that will “fundamentally transform our relationship with technology.” According to an October 2024 survey by Gartner, just four out of 123 IT leaders said Copilot provided significant value to their companies.
link? fits in my next essay
Why do so many people here post comments about articles, studies, whatever, without links?
It's weird.
This? https://www.gartner.com/en/documents/5659223
Intriguing spotlight. Ultimately, can test time compute (time * multiple output evaluations) deliver sustainable improvements.
Intriguing indeed.
Precisely…
“…whereupon I spent the rest of the evening smearing water around hundreds of plates.” 🍽️
A beautiful and memorable analogy for where LLMs are at the moment. We are left with saturated aprons at best.
This is what it means to “think” and consider” - putativus - without the application of human consciousness, character and clarity.
The wonderment may be winding down. As others just try and move stuff around and AI wash everything in its wake.
All the scaling has failed since general intelligence is just that. Factual statements by LLM are missing the human test. They must get the facts right. The missing component is explicit knowledge. Only semantic AI models (SAMs) have the ability to certify results. Here is a way that a SAM could improve training while markedly cutting costs.
http://aicyc.org/2024/10/05/llm-training-cost-reduction-using-semantic-ai-model-sam-knowledge-graph/
The problem is that nothing actually scales permanently, getting better on a linear trajectory.
Every phenomenon follows an S-Curve and plateaus eventually.
In the case of LLMs, we could see this with the data that was published in the 2022 "Emergence" paper when the graphs were replotted from the deliberately misleading false origin and log X-Axis to standard true zero-origin and standard linear scale the plateau is very clear.
Inference time and/or efficiency probably is the next scaling frontier. These models have the entire corpus of human knowledge. So the answer is in there somewhere. You just have to figure out how to pull that needle out of the proverbial haystack.
I just came across a video of a talk by Thomas Friedman is which he did a very hard sell of the idea that we will have AGI in the next few years and that it will "transform everything". I really don't understand where this is coming from. It seems to be a kind of naive (or perhaps sinister) stand in for the likely outcome: that (possibly very) useful AI will make many people more productive and that "everything" will be "transformed" only because those with power will demand that everyone do more, faster, and better — or be replaced. In other words, it sounds to me like it's a way of framing what is in fact a likely power grab by capital and elites as a kind of technological inevitability that we will just have to come to terms with, rather than something that is fundamentally political and social, that we (ought to) have the power to resist or shape to our benefit.
Scaling is religion and test time training is the second coming of Jesus Christ, here to save LLMs from original spin.
https://www.businessinsider.com/microsoft-ai-artificial-intelligence-bet-doubts-marc-benioff-satya-nadella-2024-11?utm_source=linkedin&utm_medium=social&utm_campaign=business-headline-graphic
If LLMs were only given infinite time , surely they could solve physics - and maybe even tic tac toe.
And probably the farmer/goat/cabbage/wolf river crossing problem too