The Pope appears to understand AI better than Geoffrey Hinton does.
What a thing says doesn’t tell you how it came to say it
New interview with Geoff Hinton:
I could write a long essay about why this makes very little sense, similar to the one I recently wrote about Richard Dawkins, when he made a similar error, the crux of which was below. All I would needto change is the words in bold.
The fundamental problem here is that [insert name of well-known senior figure here] doesn’t reflect on how [AI] outputs have been generated. Claude’s outputs are the product of a form of mimicry, rather than as a report of genuine internal states.
Consciousness is about internal states; the mimicry, no matter how rich, proves very little. Dawkins seems to imagine that since LLMs say things people do, they must be like people, and that simply does not follow.
In his framing, [Senior Figure X] confuses himself, and does violence to the concept of consciousness [and what it is to be a being]. You can’t just look at the outputs, without investigating the underlying mechanisms, and conclude that two entities with similar outputs reach those similar outputs by similar means. And the differences are immense; one (the LLM) effectively memorizes the entire internet; the other (the human) builds a mental model through experience with world.
But why bother to rewrite that essay?
A couple days ago, before Hinton’s interview, Pope Leo XIV said it better, and more concisely, in, of all things, a tweet (following up his excellent Encyclical):
True comprehension comes from experience, not text approximation.
§
As Valerio Capraro notes, the Pope’s point is quite similar to a point that we made together with Walter Quattrociochi in February in Nature:
§
LLM researchers are NOT creating beings.
They are creating interactive fiction that is trained to predict the language of actual beings.
Those two are NOT the same.
And Hinton should know better.
P.S. Want to know my thoughts on internal “emotion” states in LLMs, and why they are not compelling evidence? Stay tuned for a forthcoming essay. For bonus points, you can see my recent impromptu debate with the artist/musician Grimes on AI and consciousness, in two threads here and here.
How many times does it have to be said and why are allegedly intelligent people spouting such garbage? It's just fucking SOFTWARE! And you don't have to be the Pope to figure it out.
I missed that pretty sharp description by the Pope. It seems odd that Hinton would make this category error. It’s like speculating that vision models can really ‘see’ (and are thus conscious too?)
It’s useful UI as a kayfabe, but people are falling for the presentation.
Quick note- the link to your previous Richard Dawkins piece is incorrect- it links to the Pope’s statement.