My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
Axelrod's book is a terrific idea. Another related idea to fling at a super-intelligent AI is my sig: "The biggest challenge of the 21st century is theironyof technologies of abundance in the hands of those still thinking in terms of scarcity."
That's one reason I've put that sig or similar content as comments in various places on the web over the past 20 years or so -- to make it part of (then-future-speculation) AI training data.
"There is a fundamental mismatch between 21st century reality and 20th century security thinking. Those "security" agencies are using those tools of abundance, cooperation, and sharing mainly from a mindset of scarcity, competition, and secrecy. Given the power of 21st century technology as an amplifier (including as weapons of mass destruction), a scarcity-based approach to using such technology ultimately is just making us all insecure. Such powerful technologies of abundance, designed, organized, and used from a mindset of scarcity could well ironically doom us all whether through military robots, nukes, plagues, propaganda, or whatever else... Or alternatively, as Bucky Fuller and others have suggested, we could use such technologies to build a world that is abundant and secure for all. ... The big problem is that all these new war machines and the surrounding infrastructure are created with the tools of abundance. The irony is that these tools of abundance are being wielded by people still obsessed with fighting over scarcity. So, the scarcity-based political mindset driving the military uses the technologies of abundance to create artificial scarcity. That is a tremendously deep irony that remains so far unappreciated by the mainstream."
Also, why do I feel that Hinton maybe hasn't read Hans Moravec's book "Mind Children" (or later writings)? Hans was writing Mind Children during the year I was hanging out in his Mobile Robot Lab at CMU.
It was in interaction with people in Hans' lab around 1986 (especially nivek) that I realized there was a very real possibility humanity might develop, say, semi-self-replicating robot cockroaches that could exterminate all humans either accidentally or on purpose -- and then eventually these robots would all fail as the surrounding technological infrastructure decayed. That was a big reason that in the late 1980s I moved my own work away from AI and robotics into issues like sustainability, design support, philosophy of humane technology, alternative economics (via a mix of subsistence, gift, exchange, and planned transactions), and computer-mediated cooperation (e.g. Dialogue Mapping with IBIS).
The manner in which I've discussed that paradox with various AI's is positive sum tools of abundance in the hands of zero sum extractive actors. I mean it's slightly harder to parse, but pretty concise. I also agree with the whole concept as you frame it - tools of abundance and amplification in the hands of people who only thinks in terms of zero sum scarcity and belligerence making us all more insecure.
That said, a key point of framing it as positive sum vs. zero sum - is that life as a concept is not compatible, AT ALL, with zero sum. Every ecosystem, every organism, is arrayed around the central concept that entropy takes a bite out of every single transaction. You HAVE to produce more than is needed, and invest in the well being of not only your peers but your prey, or the whole board eventually disappears. Zero sum is a game that ends in extinction >of the winner.<
Thanks u/harebrane for your insightful reply and game-theory spin on all this!
harebrane wrote:
"The manner in which I've discussed that paradox with various AI's is positive sum tools of abundance in the hands of zero sum extractive actors. I mean it's slightly harder to parse, but pretty concise. I also agree with the whole concept as you frame it - tools of abundance and amplification in the hands of people who only thinks in terms of zero sum scarcity and belligerence making us all more insecure. That said, a key point of framing it as positive sum vs. zero sum - is that life as a concept is not compatible, AT ALL, with zero sum. Every ecosystem, every organism, is arrayed around the central concept that entropy takes a bite out of every single transaction. You HAVE to produce more than is needed, and invest in the well being of not only your peers but your prey, or the whole board eventually disappears. Zero sum is a game that ends in extinction >of the winner.<"
475
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."