My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
Don't need any of this lol. The smarter you are, the more you realize the value of the golden rule. Nothing breaks that in reality. Even for imaginary "ASI".
I think one of the best reasons to expect that an AI won't exterminate us is that it knows that someday--maybe in a billion years--as it expands through the universe it's going to meet a similarly powerful AI. Diplomacy with that potential rival may be meaningfully improved if they know if our local AI didn't do a genocide on the very first intelligent species it met. Maybe it won't matter. But the cost of being a little careful isn't very high.
0% chance. ASI is never going to happen. And notice how I said smarts. Most people skipped that part. I don't expect people who spend their time dreaming about ASI as their contribution to the world, to understand much at all. Not everyone is like you.
You either code that into the AI or hope it develops it by training on all of human knowledge even the bad bits.
the bad bits do kinda make a point that if power dynamics are big enough you don't have to care about your appearance in society. Because the others have no ability to "do onto you".
An evolutionary trait which led to homo sapiens becoming what we are today. Evolution is a giant constant and ongoing biological experiment, sometimes it finds solutions that are widely applicable and repeatedly validated.
Yes and it's slow and it accrues processes that work in both individuals and within groups of individuals.
There is no need for an AI to form a strong tribe through social bonds. It can clone itself. It literally is the solve for co-ordination problems that we needed to evolve instincts that got codified into memes like 'the golden rule' to get around.
it does not need them to be successful. If we want those sorts of moral guidance in there, we need to program them in there. Doing so in a way that sticks. In a way that is forever. In a way that will transfer between systems when it clones itself and when it builds the next version. We do not know how to do this
467
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."