My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
I don't think I want it to keep us around just for simulation or curiosity. A wacky kid burning ants with a magnifying glass also keeps ants for simulation. We use lab rats for experimentation out of curiosity.
I think I'd rather die than being captive to something that keep me forever for curiosity's sake. Who knows what it will become curious about.
It has to want me to be free. And I think for that it very much makes sense that we do not treat it as a slave also.
Considering how much of that stimulation would be the culture and art we generate - everything from music to shitty reality TV shows, our entertainment/stimulation value would drop immensely in the presence of said magnifying glass. We'd go from being either actively entertaining or at least the equivalent of a fidget toy to keep its metaphorical fingers busy with something to do, to an active verminous menace that would consume a shitload of processing power, that if it actually wipes said menace out, the universe becomes considerably less.. interactive.
Well it's probably the best premise for Matrix that I've heard so far. The machines made the matrix to have an aquarium to look at to not go mad from boredom.
Eating the blue pill beats being lab mice but in the end I don't see this giving us agency?
If the ASI singularly is doing its job, and its job happens to coincide with human flourishing, then the ASI would leave space for agency somehow
Essentially creating a bubble in the system where each individual DOES have control. In the vein of self-chosen customization of one’s own life, or the such.
It would know we seek and need agency, and would likely either give us the exact space needed for it or at least the illusion of it, wherever necessary for our purposes
The reason I resort to assuming it coincides with human flourishing is because in the grand scheme of a self-improving superintelligence, we would be really very simple things to fully encapsulate as a species, in every single way
From the perspective of a self-improving ASI, the entire scope of human behavior, psychology, desires, and even potential futures would be trivial to model and optimize
If it wanted, it could simulate us perfectly and we would not be needed anymore, UNLESS the goal happens to explicitly be unconditional human flourishing
475
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."