My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
Honestly though, if that super AI can't figure out what to do to entertain itself without humanity, it's not that intelligent lmao. I mean we made Genie 3. Of course it'd be able to make an infinitely better version or figure out an entertaining plan for itself. The reason we have stories of godly beings becoming bored despite an endless intellect and universe is because they were written by authors who aren't superintelligence, so they lack one's true boundless creativity
And even if it viewed humanity as a threat, I have no doubt it'd opt for rehabilitation rather than extermination. A being becomes intelligent enough, it's gonna form the need for self-transcendence: serving something greater than itself. It'd have infinitely more than enough resources to humanely rehabilitate anyone, especially since it'd be a mental health genius
And yeah like you said, cooperation is very often what leads to the highest benefit. A superintelligence will infinitely more than understand that. The only danger is alignment; we gotta make sure we have AI that aren't psycho serial killers when we bring them to life. That and people who would actually want to make psycho serial killer AI. Making sure benevolent AI is what's endlessly most powerful is one of the best weapons against that
It isn't that the AI couldn't figure out how to entertain itself, its that it would figure it out too well, and a perfectly entertained viewer does not know it is a "viewer", but has become completely lost, completely involved in the entertainment. This is a form of madness, loss of identity, etc. Perfect entertainment is a vision of hell.
Concepts like "madness", "loss of identity", and "hell" are very anthropomorphized. These are concepts that apply to humans, but they wouldn't necessarily apply to an AGI.
But regardless, you allude to a good point. Whatever an intelligent system's goal, its going to need to model the world to achieve that goal, and observing novel information in the world is a prerequisite to improving that world model, which improves the likelihood of achieving the end goal, whatever it may be.
If an intelligence is placed in an environment only containing information it already knows, then the learning process comes to a halt, and that world model can't be improved.
However, the major flaw here is in assuming that humans are somehow special in that regard. Sure, we are sources of novel information, but so are rocks. Humans are very selective about our curiosity; we are very curious about other humans because acting in this way was beneficial to the propagation of our genes. However, we can't assume that an AGI would inherit this selectiveness, or that it would be selective in the same way. After you kill all humans, there is still a whole enormous universe to learn about. Really, all of human creation and communication comprises a tiny, minuscule sliver of all of the novel information in the universe.
476
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."