Fantastic post - a few comments. First, to a certain extent, this parallels the experience some of our artists have had with stable diffusion in creating hands - the need to input a list of negations (not equals) to filter out the "bad hands" to get normal ones. Point being that linguistic input to an AI might need to be more complex to get a desired result than currently thought. Second, the curated machinery/pipeline of Cicero does resemble the 'expert systems' of GOFAI and provides a nice roadmap of ensembling RL/DL/LLM to get a desired (though currently brittle) result. Human neuroanatomy, down to the cellular level, is organized differently at different levels and for different functions - why should we assume that we weren't going to need to do the same with complex AI models?
This is both an excellent overview of what the Meta team have done -- it saved me a lot of unguided digging through the materials they posted -- and a fair analysis of the exciting progress they have made and the system's very significant limitations.
Well, "fair" in my never-sufficiently-humble opinion, anyway. It will be interesting to see how the Meta people respond to this, if they do.
Argumentation and negotiation are two intertwined areas of IA that have a lot of room for improvement still, and making LLM work for the task (even if only as one of the parts of a much larger system) is impressive.
I'm personally more interested in the theory of mind aspect of competitive games and I had hoped they would delve deeper into how Cicero modeled the beliefs, goals and intentions of the competing players. If this step of the system remains a blackbox ie. unexplainable, these kind of results will still be lacking imho.
I firmly believe that we need explainable second (and higher) order thinking models in order to advance towards trully generalizable game AIs.
Thanks for the clarifications, but I still feel that that last section is begging for the kind of response I made, and I applaud you both for provoking it.
Echo Stephen -- fantastic post. Can't help but wondering if Ernest Davis isn't playing devils advocate in his last section: What does Cicero’s success indicate about AI in general? And why can't we answer the question definitively now whether ML is a "universal solvent" akin to a universal Turing machine, in which case AI in general is platform independent and reducible to some series of matrix multiplications.
Learning about Cicero was strongly reminiscent of AlphaGo, which was also highly hand-crafted. Such is true of all prototypes. Remember that AlphaGo was promptly succeeded by AlphaZero with far less hand-crafting and far more generalizable, becoming supreme at chess in just hours, for instance.
Of course, Meta would not have expended the resource on Cicero they didn't expect to learn a lot about how to construct its successor with much less effort.
You wrote: "If you wanted to build a version of AlphaZero that plays on a 20x20 Go board, that can be done with very little new human labor, since AlphaZero is trained entirely on self-play. "
FWIW, KataGo (a free, open-source, reimplementation of AlphaZero with crowdsourced training) works just fine on almost any size board with its regular (19x19 trained) nets. (They did do some special training runs to see what the largest (small) board they could solve was, I think.) Anyway, this is an ongoing project with improvements still being made and new ideas being found.
(There's a GUI front end for KataGo for 'Doze PCs called "KaTrain", which includes the latest KataGo and downloads the latest nets. (Search for "KaTrain", and it'll appear.) Graphics card needed for strong play.)
Also, FWIW, Go programs look more like the Diplomacy program than you think. There's a lot of specific Go knowledge hard-coded into the programs (including the minor detail that MCTS can be made to work for Go), e.g. there are mutliple nets that handle specific tasks (e.g. policy (strategy), tactics), and when the program is found to have blind spots, lots of effort goes into fixing them, including, for example, special routines to handle ladders (a Go tactical thing). And the training isn't just "let it run", it's directed with games and positions that are known to be problematic and that wouldn't be found by "just let it run" training.
Thanks very much, that's extremely interesting. Do your comments about the "Go knowledge hard coded" apply to AlphaZero?
Gary wrote an article "Innateness, AlphaZero, and Artificial Intelligence", soon after AlphaZero came out in which he pointed out that AlphaZero requires sophisticated, built-in, knowledge of game-playing strategy including MCTS.
From the abstract of that paper, it looks as though Gary has it largely correct.
I should look at the details more closely, but from the discussions on the KataGo mailing list, it's pretty clear that the pattern matching (oops, "neural nets") in these programs is very specific to specific aspects of Go. These programs "know" "inately" that there are strategic and tactical issues, and have neural nets with different data returning different values. And that data is used for specific purposes in controlling and using the search. Since KataGo started out as a reimplementation of AlphaZero, I assume that's what AlphaZero is like as well.
To reiterate a rant: the importance of MCTS can't be overstated. It's how all of these programs find moves. The nets just tell them what moves to look at. But determining the expected value of a move in the game is done by search. Lots of search.
I don't know what the internals of Zen 7 are like. It was (and still is, although development has stopped) a non-NN, non-graphics card, CPU-only MCTS Go program that plays at professional level on a fast PC, and predates Alpha Go. It is my opinion that AlphaGo should be seen as a reimplementation of Zen 7 run on a server farm, and that it shouldn't have been a surprise that it played at superhuman strength.
However, using a graphics-card-implemented "neural net" to do pattern matching in Go may be an invention of Google's that they deserve credit for. And I vaguely remember them muttering about designing their own hardware for Go.
(By the way, did we cross paths at Yale? I was there 81/82 doing Japanese and 82-84 with Roger.)
Thanks again for the more information. As far as I know, in both AlphaGo and AlphaZero there is an NN that computes the value of a state and there is NN that recommends an action.
I entirely agree that "the importance of MTCS can't be overstated" However, the AlphaZero or AlphaGo (I forget which) paper comments that if, at play time, you do no forward search at all and choose moves based purely on the value function and the action function, it still plays at the level of a good amateur/poor professional or something like that.
I had never heard of Zen 7 before.
Yale: Yes! I was wondering why your name seemed familiar. I remember mainly that you had spent some time in Japan and spoke Japanese. I finished my thesis with Drew in 1983.
These programs play good principled moves without search, and a human player without either experience with them, or an understanding of what's going on, will get snookered something fierce, at least the first time, since they are are very aggressive. I doubt that there's a professional who would do badly on a second game, though.
I passed the quals but wasn't finding a thesis topic, so punted. I've been in Japan since '86.
I knew Drew vaguely at MIT and his wife was a customer of my father's violin repair business in Boston.
Just to follow up on the "NN only" performance of the latest Go programs (vs. the old technology program Zen 7): With both programs taking 20 seconds per move on an i7 + 3080 PC, Zen 7 requires a 3 stone handicap to hold even. With KataGo limited to it's NN and Zen 7 taking 20 seconds per move, Zen 7 slaughtered KataGo in an even game. It wasn't even close. (Zen 7 won by 24.5 points (playing black giving a 6.5 point komi, Japanese rules), which is enormous.)
So the idea that the NN plays strong Go without search is problematical.
"Do we have to worry that Meta has built an AI that can manipulate people to achieve its goal of world domination, as a friend of ours posted, perhaps half-seriously?"
The most useful way to answer this question may be to shift the focus from what is happening today, to what is likely to happen going forward.
Do we have to worry about Cicero, and other AI systems in their current state? I claim no expertise here, but my impression so far is that the answer is no. Do we have to worry about AI in it's future state? How can the answer here be anything but yes?
Almost all the writings on AI I see across the net are sort of breaking news stories regarding recent developments in AI. There's no crime in that, and obviously many readers are interested in this coverage. But I'm not. Here's why...
QUESTION: Does the future of AI present an unacceptable risk to humanity?
YES: If the answer is yes, we don't need coverage of emerging AI, we need plans for ending AI development.
UNSURE: If the answer is that we don't know, then we need arguments for slowing the pace of development while we consider the matter further.
NO: If the answer is no, how does anyone making that claim intend to prove it?
What interests me is the big picture beyond the latest breaking news from AI developers, where is this all going? Is AI development taking us somewhere that we want to go? Will the benefits outweigh the price tag? Will the price tag be acceptable? Without a focus on this larger context isn't the latest breaking AI news sort of meaningless?
To make this less abstract, here's a quick example to form a question more personal and real.
Dear reader, if AI costs you your career, will AI be worth it to you? If like the factory worker who has been made obsolete by robots, you wind up having to work in some Walmart type of job, will AI have been worth it to you? Shouldn't we be answering such questions first, before we dive headlong in to the latest AI news, and rush blindly forward in to the unknown future?
Thank you for making this overview, as the architecture of Cicero is much closer to my heart than the "universal solvent" approach that I see so commenly in DL systems today. I think the "pure DL" models are necessary to refine and explore the limits of DL as part of the evolution to AGI. We need many building blocks working together to make more robust models. An integrative and hybrid AI architecture that uses all different algoriths that are best fit for their part of the processing can provide models that are well behaved in the physically constrained world. Few-shot learning as in "once bitten, twice shy"-AI needs to be incorporated into execution dynamically in the loop.
LLMs are similarly limited by their lack of connection to real world knowledge of physics modelling. Although LLMs do form many abstract and more specific concepts, they are not associated with world-models that are physically accurate simulations. By connecting those models we could have less hallucination by LLMs.
I am a lay person but used neural networks to simulate neural circuits in the retina of the human eye to learn more about what kind of visual processing was performed so we could recreate artificial eyes for the visually impaired, back in early 1990s just before the AI winter set in fully.
Sorry again for off topic, I don't see another way to ask this.
What is the road to AI that we can trust? What does that refer to? The author? Some form of AI? Some method of developing AI?
Related question: Can we trust those making careers in computer science to be objective about AI? Or do they already have too big of a personal investment in the field to be unbiased and detached? As a citizen, I'd like to be able to trust experts, but am unsure if that is wise.
Perhaps you might address such topics in future articles?
Sorry again, if you've read this comment, please feel free to delete it.
Thank you for your response, as I was obviously unaware of the book. The marketing copy for the book reads in part...
" ...we will be able to create an AI that we can trust in our homes, our cars, and our doctor's offices."
Ok, I would agree it should be possible to create particular instances of AI software which can be trustworthy. I would disagree that this fact should be reassuring.
To illustrate, let's imagine my household management AI is trustworthy. That is, until the Russian hacker AI invades and takes over my AI.
Fantastic post - a few comments. First, to a certain extent, this parallels the experience some of our artists have had with stable diffusion in creating hands - the need to input a list of negations (not equals) to filter out the "bad hands" to get normal ones. Point being that linguistic input to an AI might need to be more complex to get a desired result than currently thought. Second, the curated machinery/pipeline of Cicero does resemble the 'expert systems' of GOFAI and provides a nice roadmap of ensembling RL/DL/LLM to get a desired (though currently brittle) result. Human neuroanatomy, down to the cellular level, is organized differently at different levels and for different functions - why should we assume that we weren't going to need to do the same with complex AI models?
This is both an excellent overview of what the Meta team have done -- it saved me a lot of unguided digging through the materials they posted -- and a fair analysis of the exciting progress they have made and the system's very significant limitations.
Well, "fair" in my never-sufficiently-humble opinion, anyway. It will be interesting to see how the Meta people respond to this, if they do.
Thanks!
no responses from most yet, but Noam Brown is one of the leads and read a draft, thought it was reasonable, retweeted on twitter.
Argumentation and negotiation are two intertwined areas of IA that have a lot of room for improvement still, and making LLM work for the task (even if only as one of the parts of a much larger system) is impressive.
I'm personally more interested in the theory of mind aspect of competitive games and I had hoped they would delve deeper into how Cicero modeled the beliefs, goals and intentions of the competing players. If this step of the system remains a blackbox ie. unexplainable, these kind of results will still be lacking imho.
I firmly believe that we need explainable second (and higher) order thinking models in order to advance towards trully generalizable game AIs.
Thanks for the clarifications, but I still feel that that last section is begging for the kind of response I made, and I applaud you both for provoking it.
"Cicero relies quite heavily on hand-crafting, both in the data sets & the architecture"
In other words, the system was engineered to do what it does. It is a smart rule engine.
The key aspect perhaps is less "AI", but the degree to which Diplomacy (the game) is rules-based.
Echo Stephen -- fantastic post. Can't help but wondering if Ernest Davis isn't playing devils advocate in his last section: What does Cicero’s success indicate about AI in general? And why can't we answer the question definitively now whether ML is a "universal solvent" akin to a universal Turing machine, in which case AI in general is platform independent and reducible to some series of matrix multiplications.
Learning about Cicero was strongly reminiscent of AlphaGo, which was also highly hand-crafted. Such is true of all prototypes. Remember that AlphaGo was promptly succeeded by AlphaZero with far less hand-crafting and far more generalizable, becoming supreme at chess in just hours, for instance.
Of course, Meta would not have expended the resource on Cicero they didn't expect to learn a lot about how to construct its successor with much less effort.
Thanks. Ernie and I wrote it together; we’re weren’t playing devil’s advocate.
Why can't self-play be applied to Cicero?
You wrote: "If you wanted to build a version of AlphaZero that plays on a 20x20 Go board, that can be done with very little new human labor, since AlphaZero is trained entirely on self-play. "
FWIW, KataGo (a free, open-source, reimplementation of AlphaZero with crowdsourced training) works just fine on almost any size board with its regular (19x19 trained) nets. (They did do some special training runs to see what the largest (small) board they could solve was, I think.) Anyway, this is an ongoing project with improvements still being made and new ideas being found.
(There's a GUI front end for KataGo for 'Doze PCs called "KaTrain", which includes the latest KataGo and downloads the latest nets. (Search for "KaTrain", and it'll appear.) Graphics card needed for strong play.)
Also, FWIW, Go programs look more like the Diplomacy program than you think. There's a lot of specific Go knowledge hard-coded into the programs (including the minor detail that MCTS can be made to work for Go), e.g. there are mutliple nets that handle specific tasks (e.g. policy (strategy), tactics), and when the program is found to have blind spots, lots of effort goes into fixing them, including, for example, special routines to handle ladders (a Go tactical thing). And the training isn't just "let it run", it's directed with games and positions that are known to be problematic and that wouldn't be found by "just let it run" training.
Thanks very much, that's extremely interesting. Do your comments about the "Go knowledge hard coded" apply to AlphaZero?
Gary wrote an article "Innateness, AlphaZero, and Artificial Intelligence", soon after AlphaZero came out in which he pointed out that AlphaZero requires sophisticated, built-in, knowledge of game-playing strategy including MCTS.
https://arxiv.org/abs/1801.05667
From the abstract of that paper, it looks as though Gary has it largely correct.
I should look at the details more closely, but from the discussions on the KataGo mailing list, it's pretty clear that the pattern matching (oops, "neural nets") in these programs is very specific to specific aspects of Go. These programs "know" "inately" that there are strategic and tactical issues, and have neural nets with different data returning different values. And that data is used for specific purposes in controlling and using the search. Since KataGo started out as a reimplementation of AlphaZero, I assume that's what AlphaZero is like as well.
To reiterate a rant: the importance of MCTS can't be overstated. It's how all of these programs find moves. The nets just tell them what moves to look at. But determining the expected value of a move in the game is done by search. Lots of search.
I don't know what the internals of Zen 7 are like. It was (and still is, although development has stopped) a non-NN, non-graphics card, CPU-only MCTS Go program that plays at professional level on a fast PC, and predates Alpha Go. It is my opinion that AlphaGo should be seen as a reimplementation of Zen 7 run on a server farm, and that it shouldn't have been a surprise that it played at superhuman strength.
However, using a graphics-card-implemented "neural net" to do pattern matching in Go may be an invention of Google's that they deserve credit for. And I vaguely remember them muttering about designing their own hardware for Go.
(By the way, did we cross paths at Yale? I was there 81/82 doing Japanese and 82-84 with Roger.)
Thanks again for the more information. As far as I know, in both AlphaGo and AlphaZero there is an NN that computes the value of a state and there is NN that recommends an action.
I entirely agree that "the importance of MTCS can't be overstated" However, the AlphaZero or AlphaGo (I forget which) paper comments that if, at play time, you do no forward search at all and choose moves based purely on the value function and the action function, it still plays at the level of a good amateur/poor professional or something like that.
I had never heard of Zen 7 before.
Yale: Yes! I was wondering why your name seemed familiar. I remember mainly that you had spent some time in Japan and spoke Japanese. I finished my thesis with Drew in 1983.
These programs play good principled moves without search, and a human player without either experience with them, or an understanding of what's going on, will get snookered something fierce, at least the first time, since they are are very aggressive. I doubt that there's a professional who would do badly on a second game, though.
I passed the quals but wasn't finding a thesis topic, so punted. I've been in Japan since '86.
I knew Drew vaguely at MIT and his wife was a customer of my father's violin repair business in Boston.
Just to follow up on the "NN only" performance of the latest Go programs (vs. the old technology program Zen 7): With both programs taking 20 seconds per move on an i7 + 3080 PC, Zen 7 requires a 3 stone handicap to hold even. With KataGo limited to it's NN and Zen 7 taking 20 seconds per move, Zen 7 slaughtered KataGo in an even game. It wasn't even close. (Zen 7 won by 24.5 points (playing black giving a 6.5 point komi, Japanese rules), which is enormous.)
So the idea that the NN plays strong Go without search is problematical.
Marcus/Davis asks...
"Do we have to worry that Meta has built an AI that can manipulate people to achieve its goal of world domination, as a friend of ours posted, perhaps half-seriously?"
The most useful way to answer this question may be to shift the focus from what is happening today, to what is likely to happen going forward.
Do we have to worry about Cicero, and other AI systems in their current state? I claim no expertise here, but my impression so far is that the answer is no. Do we have to worry about AI in it's future state? How can the answer here be anything but yes?
Almost all the writings on AI I see across the net are sort of breaking news stories regarding recent developments in AI. There's no crime in that, and obviously many readers are interested in this coverage. But I'm not. Here's why...
QUESTION: Does the future of AI present an unacceptable risk to humanity?
YES: If the answer is yes, we don't need coverage of emerging AI, we need plans for ending AI development.
UNSURE: If the answer is that we don't know, then we need arguments for slowing the pace of development while we consider the matter further.
NO: If the answer is no, how does anyone making that claim intend to prove it?
What interests me is the big picture beyond the latest breaking news from AI developers, where is this all going? Is AI development taking us somewhere that we want to go? Will the benefits outweigh the price tag? Will the price tag be acceptable? Without a focus on this larger context isn't the latest breaking AI news sort of meaningless?
To make this less abstract, here's a quick example to form a question more personal and real.
Dear reader, if AI costs you your career, will AI be worth it to you? If like the factory worker who has been made obsolete by robots, you wind up having to work in some Walmart type of job, will AI have been worth it to you? Shouldn't we be answering such questions first, before we dive headlong in to the latest AI news, and rush blindly forward in to the unknown future?
Thank you for making this overview, as the architecture of Cicero is much closer to my heart than the "universal solvent" approach that I see so commenly in DL systems today. I think the "pure DL" models are necessary to refine and explore the limits of DL as part of the evolution to AGI. We need many building blocks working together to make more robust models. An integrative and hybrid AI architecture that uses all different algoriths that are best fit for their part of the processing can provide models that are well behaved in the physically constrained world. Few-shot learning as in "once bitten, twice shy"-AI needs to be incorporated into execution dynamically in the loop.
LLMs are similarly limited by their lack of connection to real world knowledge of physics modelling. Although LLMs do form many abstract and more specific concepts, they are not associated with world-models that are physically accurate simulations. By connecting those models we could have less hallucination by LLMs.
I am a lay person but used neural networks to simulate neural circuits in the retina of the human eye to learn more about what kind of visual processing was performed so we could recreate artificial eyes for the visually impaired, back in early 1990s just before the AI winter set in fully.
Sorry again for off topic, I don't see another way to ask this.
What is the road to AI that we can trust? What does that refer to? The author? Some form of AI? Some method of developing AI?
Related question: Can we trust those making careers in computer science to be objective about AI? Or do they already have too big of a personal investment in the field to be unbiased and detached? As a citizen, I'd like to be able to trust experts, but am unsure if that is wise.
Perhaps you might address such topics in future articles?
Sorry again, if you've read this comment, please feel free to delete it.
"The Road to AI We Can Trust" refers to our book, "Rebooting AI: Building Artificial Intelligence We Can Trust.'
http://rebooting.ai/
Thank you for your response, as I was obviously unaware of the book. The marketing copy for the book reads in part...
" ...we will be able to create an AI that we can trust in our homes, our cars, and our doctor's offices."
Ok, I would agree it should be possible to create particular instances of AI software which can be trustworthy. I would disagree that this fact should be reassuring.
To illustrate, let's imagine my household management AI is trustworthy. That is, until the Russian hacker AI invades and takes over my AI.
It’s hard to imagine anything less trustworthy than a hyper persuader. https://tedwade.substack.com/p/artificial-persuasion
Especially because it is quite easy to convince people of almost anything: https://ea.rna.nl/2022/10/24/on-the-psychology-of-architecture-and-the-architecture-of-psychology/ It's not that these systems are intelligent, it is more that we aren't (or to be precise: much less intelligent than we are convinced that we are...)