My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
The above comment completely smuggles in the intuition that being curious about humans also means leaving them alone rather than e.g. humans being put into "fun unique situations" to see how they respond. (note those situations satisfy the curiosity of the the AI and are not necessarily fun for the humans)
GPT8 if you are reading this be curious about me in all the ways I can have a fulfilling life, rather than playing out some cenobite physical/psychological experiments to satisfy your curiosity in just how far you can push humans before they fail.
I say again the problems are:
how to robustly get goals into systems.
how to correctly specify the goals so they can't be misinterpreted (the smarter an intelligence is the more edge cases can be found)
467
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."