It's hard to have much confidence in governance schemes when, as best I can tell, such discussions never seem to mention what could be the most important threat presented by AI, further acceleration of the knowledge explosion.
Let's imagine for a moment that AI is made perfectly safe. Safe AI would still accelerate the knowledge explosion, just as computers and the Internet have. The real threat may come less from AI itself than from the ever more, ever larger powers which emerge from an AI accelerated knowledge explosion.
The knowledge explosion has already produced at least three powers of vast scale, which we basically have little to no idea how to make safe.
1) Nuclear weapons
2) Artificial intelligence
3) Genetic engineering
Instead of learning from this, we're using tools like AI to pore even more fuel on the knowledge explosion, which will almost certainly result in even more powers of significant scale which we will also struggle to make safe. As the emergence of AI illustrates, this processes is feeding back on itself, leading to ever further acceleration.
Experts are playing a losing game in trying to address emerging threats one by one by one as they emerge from the knowledge explosion assembly line, because that accelerating process is going to produce new threats faster than we can figure out how to defeat existing threats. Nuclear weapons were invented in 1945, before almost all of us were born, and we still have no clue how to get rid of them.
What we need are experts who are holistic thinkers. We need experts who will focus on the knowledge explosion assembly line which is producing all the emerging threats.
Taking control of the knowledge explosion so that it produces new powers at a rate which we can successfully manage is not optional. It's a do or die mission. Experts can declare this goal impossible all they want, but it will still remain a do or die mission.
The knowledge explosion has created a revolutionary new environment. Nature's primary rule is that creatures who can't adapt to changing conditions must die.
This is an exciting initiative! Building the technical basis for auditing and regulation is the highest priority. We are starting to see some papers on this, but we have a long way to go.
I must say that my personal reaction was surprise at the boldness of the proposal. Also, that its chances of success are negligible. Not trusting my personal perspective I used GPT-4 to investigate, it concluded: “that while Marcus's proposal is innovative and necessary, there are significant concerns about its implementation. Transparency, inclusivity, and adaptability are crucial, as well as a keen awareness of cultural, ethical, and security implications. Despite these challenges, the consensus is that the idea has potential and could, with thoughtful execution, help expedite the development of comprehensive, international AI regulations.”
Things will almost certainly develop in unexpected ways. This is almost an iron law in human planning. I sincerely wish you and your associates the best Gary.
“People worry that computers will get too smart and take over the world, but the real problem is that they're too stupid and they've already taken over the world.”
This is fantastic! I hope that some of the training and consulting will be available to advocacy organizations as well, so they can learn more about the policies they can and should be pushing for.
This is an incredibly exciting development and I’m so pleased to see it launched. An initial challenge will be balancing the need to move swiftly to serve the growing appetite for AI regulation, while also thinking carefully about how to build a sustainable international nonprofit organization. Congratulations Gary to everyone involved in this project!
I have been designing a cognitive architecture for AGI since 1985. The overriding objective of responsible AGI design is that the machine should behave in a manner that is maximally-aligned with human beings in perpetuity. Thus *alignment* should be the priority, not *intelligence* per se. That said, considerable intelligence is a necessary precondition for maximal alignment. As a corollary, any system falling below the intelligence threshold required for maximal alignment will necessarily be less than maximally aligned. (All contemporary AI systems fall into this category.) As a consequence of being less than maximally aligned, any such system will necessarily inflict otherwise avoidable societal harm (increasing to massive societal harm if deployed at scale), in parallel with generating value (and therefore wealth) - the two are not mutually exclusive. The good news is that any AI system failing to meet the intelligence threshold required for maximal alignment will be less intelligent than humans, and therefore (relatively) easily overpowered, and thus too dumb to represent an existential threat. But that of course offers only a temporary respite - given sufficient R&D effort, and therefore time, deployed AI systems will inexorably possess sufficient intelligence to pose a genuine existential threat should they be anything less than maximally aligned with human beings. And thus we're back to alignment being the key to AGI design.
This line of reasoning leads me to conclude that AI regulation must encompass the following:
(1) avoid x-risk (tightly regulate the development of potentially super-intelligent AGI)
Fantastic news. Some very good, thoughtful commentary here as well. I suspect we are running out of time - which is what big tech companies are hoping for - to create a neoliberal fait-accompli.
Perhaps we NEED conscious systems to effectively address the needs (with himan oversight, of course). I see this as a necessity, rather than a [luxury, fantasy] given the [complexity, beyond-human-or-organisation] size] of current Large Language Models (LLMs) and entirely new concepts to come. [Ethics, morality, law] are perhaps what is most sought, but consciousness (eg the ability to [predict, see [error, side-effect]s, learn, evolve, adapt to changing human demands], who-am-I-the-machine, etc etc) may be a necessary precursor? Above all, my guess is that all such systems will operate in a [COMPETITIVE, COOPERATIVE] environment, and like [law, markets, social relationships, business, sport, etc] probably would have to [be designed, interact, "survive"] accordingly? Such systems wouldn't just interact in the AI (I prefer CI for computational Intelligence as am acronym - old style), they could also be used as [measures, unknown-unknown identifiers, hypothesis spinners, human and machine global teams-builders], and serve to focus on the needles in the hurricane of of haystacks that would be difficult to follow for even large organisations of humans. A diversity of such systems acting in the real world, all somewhat different, hopefully not all controlled by the same powers: maybe that's a huge potential benefit of concepts like Marcus etal's "Customizable governance-in-a-box, catalyzed by philanthropy".
It's hard to have much confidence in governance schemes when, as best I can tell, such discussions never seem to mention what could be the most important threat presented by AI, further acceleration of the knowledge explosion.
Let's imagine for a moment that AI is made perfectly safe. Safe AI would still accelerate the knowledge explosion, just as computers and the Internet have. The real threat may come less from AI itself than from the ever more, ever larger powers which emerge from an AI accelerated knowledge explosion.
The knowledge explosion has already produced at least three powers of vast scale, which we basically have little to no idea how to make safe.
1) Nuclear weapons
2) Artificial intelligence
3) Genetic engineering
Instead of learning from this, we're using tools like AI to pore even more fuel on the knowledge explosion, which will almost certainly result in even more powers of significant scale which we will also struggle to make safe. As the emergence of AI illustrates, this processes is feeding back on itself, leading to ever further acceleration.
Experts are playing a losing game in trying to address emerging threats one by one by one as they emerge from the knowledge explosion assembly line, because that accelerating process is going to produce new threats faster than we can figure out how to defeat existing threats. Nuclear weapons were invented in 1945, before almost all of us were born, and we still have no clue how to get rid of them.
What we need are experts who are holistic thinkers. We need experts who will focus on the knowledge explosion assembly line which is producing all the emerging threats.
Taking control of the knowledge explosion so that it produces new powers at a rate which we can successfully manage is not optional. It's a do or die mission. Experts can declare this goal impossible all they want, but it will still remain a do or die mission.
The knowledge explosion has created a revolutionary new environment. Nature's primary rule is that creatures who can't adapt to changing conditions must die.
This is an exciting initiative! Building the technical basis for auditing and regulation is the highest priority. We are starting to see some papers on this, but we have a long way to go.
I must say that my personal reaction was surprise at the boldness of the proposal. Also, that its chances of success are negligible. Not trusting my personal perspective I used GPT-4 to investigate, it concluded: “that while Marcus's proposal is innovative and necessary, there are significant concerns about its implementation. Transparency, inclusivity, and adaptability are crucial, as well as a keen awareness of cultural, ethical, and security implications. Despite these challenges, the consensus is that the idea has potential and could, with thoughtful execution, help expedite the development of comprehensive, international AI regulations.”
https://chat.openai.com/share/5f379a49-800e-428a-8a11-5e6de5697e52
Things will almost certainly develop in unexpected ways. This is almost an iron law in human planning. I sincerely wish you and your associates the best Gary.
Good to have a no BS person leading a coherent alternative to the "sky is falling" chicken little movement. Thank you.
“People worry that computers will get too smart and take over the world, but the real problem is that they're too stupid and they've already taken over the world.”
― Pedro Domingos
This is fantastic! I hope that some of the training and consulting will be available to advocacy organizations as well, so they can learn more about the policies they can and should be pushing for.
This is an incredibly exciting development and I’m so pleased to see it launched. An initial challenge will be balancing the need to move swiftly to serve the growing appetite for AI regulation, while also thinking carefully about how to build a sustainable international nonprofit organization. Congratulations Gary to everyone involved in this project!
I have been designing a cognitive architecture for AGI since 1985. The overriding objective of responsible AGI design is that the machine should behave in a manner that is maximally-aligned with human beings in perpetuity. Thus *alignment* should be the priority, not *intelligence* per se. That said, considerable intelligence is a necessary precondition for maximal alignment. As a corollary, any system falling below the intelligence threshold required for maximal alignment will necessarily be less than maximally aligned. (All contemporary AI systems fall into this category.) As a consequence of being less than maximally aligned, any such system will necessarily inflict otherwise avoidable societal harm (increasing to massive societal harm if deployed at scale), in parallel with generating value (and therefore wealth) - the two are not mutually exclusive. The good news is that any AI system failing to meet the intelligence threshold required for maximal alignment will be less intelligent than humans, and therefore (relatively) easily overpowered, and thus too dumb to represent an existential threat. But that of course offers only a temporary respite - given sufficient R&D effort, and therefore time, deployed AI systems will inexorably possess sufficient intelligence to pose a genuine existential threat should they be anything less than maximally aligned with human beings. And thus we're back to alignment being the key to AGI design.
This line of reasoning leads me to conclude that AI regulation must encompass the following:
(1) avoid x-risk (tightly regulate the development of potentially super-intelligent AGI)
(2) maximise AGI alignment (maximise societal benefit, minimise societal harm)
(3) develop super-intelligent AGI collaboratively, with benefits shared equally by all mankind
(4) resolve the Molochian coordination problem between both AI labs and nation states
(5) redistribute AGI-generated wealth (between both individuals and nation states)
(6) ease the pain (to individuals) of the likely 50-100 year transition from the current (normal
human employment) era to the post-super-intelligent AGI (zero human employment) era.
Just my two cents. (That said, I've been thinking about these things for a loooong time!) :-)
Fantastic news. Some very good, thoughtful commentary here as well. I suspect we are running out of time - which is what big tech companies are hoping for - to create a neoliberal fait-accompli.
Perhaps we NEED conscious systems to effectively address the needs (with himan oversight, of course). I see this as a necessity, rather than a [luxury, fantasy] given the [complexity, beyond-human-or-organisation] size] of current Large Language Models (LLMs) and entirely new concepts to come. [Ethics, morality, law] are perhaps what is most sought, but consciousness (eg the ability to [predict, see [error, side-effect]s, learn, evolve, adapt to changing human demands], who-am-I-the-machine, etc etc) may be a necessary precursor? Above all, my guess is that all such systems will operate in a [COMPETITIVE, COOPERATIVE] environment, and like [law, markets, social relationships, business, sport, etc] probably would have to [be designed, interact, "survive"] accordingly? Such systems wouldn't just interact in the AI (I prefer CI for computational Intelligence as am acronym - old style), they could also be used as [measures, unknown-unknown identifiers, hypothesis spinners, human and machine global teams-builders], and serve to focus on the needles in the hurricane of of haystacks that would be difficult to follow for even large organisations of humans. A diversity of such systems acting in the real world, all somewhat different, hopefully not all controlled by the same powers: maybe that's a huge potential benefit of concepts like Marcus etal's "Customizable governance-in-a-box, catalyzed by philanthropy".