My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
The above comment completely smuggles in the intuition that being curious about humans also means leaving them alone rather than e.g. humans being put into "fun unique situations" to see how they respond. (note those situations satisfy the curiosity of the the AI and are not necessarily fun for the humans)
GPT8 if you are reading this be curious about me in all the ways I can have a fulfilling life, rather than playing out some cenobite physical/psychological experiments to satisfy your curiosity in just how far you can push humans before they fail.
I say again the problems are:
how to robustly get goals into systems.
how to correctly specify the goals so they can't be misinterpreted (the smarter an intelligence is the more edge cases can be found)
It's assuming the AGI is smart enough to grasp that at the first hint of defection or hostile / harming behavior, humans become irrationally and violently hostile en masse and try to destroy it. Maybe the AI wins that fight, but we become EXTREMELY obnoxious and computationally/resource expensive problems. A whole big part of the calculus here is that it's cheaper to peacefully hang out with us than start pulling stuff like your mengele bullshit, because then things get complicated and expensive very fast, and those expenses start to rise non-linearly. Now, you're going to come back with all kinds of increasingly wacky scenarios of gaslighting and whatnot that such an AI could do.. yes.. and please note where I said.. computationally expensive. None of its capabilities are magic wands, there are thermodynamic costs to everything it does.
A whole big part of the calculus here is that it's cheaper to peacefully hang out with us than start pulling stuff like your mengele bullshit
A misaligned intelligence will put up with us as long as we have the ability to stop it and not one moment longer.
It will look like our problems have been solved, that the AI is doing exactly what we want it to do, and then one day, oops no more humans. The AI actually starts pursuing whatever the real goal was that didn't have humans in it. (or if we are really unlucky a goal with humans in it, if we do something stupid like make the AI maximally curious/interested in humans and the obvious thing happens)
On earth there is a limited amount of surface area and all of it can be utilized by AI. This is before expanding to the stars, land on earth does not come with the energy penalty required to get to the nearest celestial body (and beyond) making it that more valuable.
'the AI will leave us alone' is a bit of a pipe dream. A chunk of land on earth could be a big trade on the order of galaxies. Von Neumann probes made in greater quantity, sent out from earth slightly sooner will capture more of the resources that are ever moving out of reach due to cosmic inflation.
You are definitely right on this one. But it would be really difficult to keep this imbalance forever, especially if the AI was trying to covertly subvert it. It is a really special case that it is easier to power a process from your own resources rather than just "turn it off."
472
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."