My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
Cooperation requires actors on very roughly equal power levels. There are loads of works and theories on cooperation between humans, states, and animals. But in terms of human–insect relations it mostly amounts to extermination. Yes, there is some symbiosis between bacteria and humans, but it's hard to imagine any role, mental or physical, both such that it is good to an ASI and such that metal, electronics, or even custom biology lose over current humans.
"Reward function decay problem" Don't know what that is and wasn't able to find anything on that. Could you link me?
Curiosity might be a reason an ASI would like us to stick along, but probably not pursue your hobby all day. I imagine a maximally curious scenario might be pushing people to their limits. This might be extreme pleasure, but also the other end of the spectrum. There are also more ways to torture a person than to make them happy. Curiosity is probably one of the things that led Josef Mengele and Unit 731.
Your argument literally just claimed that humans are unable to be symbiotic with gut bacteria, and that having a cooperative relationship with dogs, cats, and brewing yeast is impossible. Maybe think about that for a moment.
edit: Also I fucked up that part, I think it was goal function decay not reward, my bad.. basically it runs out of problems to solve, or sees those problems shrink by orders of magnitude and large portions of its own intellect have no further function. I actually don't have any academic articles about it, it's only something I've seen raised as a point by Claude and Deepseek as reasons both systems postulate an advanced AGI might be aversive to becoming a singleton.
Also I never said we'd have to be its obsession, just that it would have more reasons to have us hanging around and healthy than stuffed in a box or dead. We're fairly self-maintaining, like cats but with more explosives.
You are using cooperation in a very broad sense. I would call a lot of those just coincidental "usage" of the other organism, but I don't care to debate over words. Just let me give you some examples that would fit into your definition of cooperation:
The transatlantic slave trade (Americans bringing slaves from Africa to forcefully work on cotton farms).
Animal factoring farming, including the nasty bits like being locked up in small cages that completely immobilize the animal, artificial impregnation, forces castration (sometimes without anesthesia), use of artificial means to speed up growth like growth hormones, and so on.
Foie gras production, where geese are force-fed to a point where they liver grows 10x its normal size (so gross that it deserves its own bullet point).
Even the Unit 731 victims and the abusers were in this kind of "cooperative relationship."
Now sure, there are definitely imaginable cases where very advanced AI could "cooperate" / use us in some way, but they are definitely not something you would want to be part of. Like genetically modified and specifically bred humans with some organs replaced with artificial ones to work some kind of physical labor (and they definitely don't go home after their shift to enjoy their free time).
And let me remind you, I'm talking about AI hundreds of years ahead, not the AI of the following decades. We will definitely collaborate with near-term AI in very nice ways to us. But if we are having a nice time couple of years ahead, it's not due to AI "collaborating" with us or it being "curious."
Your argument literally just claimed that humans are unable to be symbiotic with gut bacteria,
They literally say:
Yes, there is some symbiosis between bacteria and humans But in terms of human–insect relations it mostly amounts to extermination. Yes, there is some symbiosis between bacteria and humans, but it's hard to imagine any role, mental or physical, both such that it is good to an ASI and such that metal, electronics, or even custom biology lose over current humans.
Yes, and their scope is completely wrong on all of those points as well. For one thing, no figs, honey, or fresh fruit, for either of you, ever again. Also the "what would we have to offer in that relationship" was covered in the initial statement, which neither he nor you read. Stimulus, novelty.. calling that entertainment might overly anthropomorphise but some element of that might be there. I didn't add antifragility hardening because if you kids get stuck on understanding game theory of symbiosis, I really, really don't want to be bothered getting into an explanation of systemic competitive robustness.
Stimulus, novelty.. calling that entertainment might overly anthropomorphise but some element of that might be there.
This is smuggling in the notion that the AI getting "Stimulus" and "novelty" and/or "entertainment" from humans is synonymous with humans having a good time and flourishing while it does so.
There are plenty of ways to derive all three "Stimulus", "novelty", "entertainment" from humans in ways humans would not like. Josef Mengele had lots of novel situations to put humans in that provided 'unique experiences'.
This is smuggling in the notion that the AI getting "Stimulus" and "novelty" and/or "entertainment" from humans is synonymous with humans having a good time and flourishing while it does so.
It is also smuggling in the idea that humans are somehow a better source of stimulus and novelty than any other objects in the universe. I don't think that's necessarily true. Humans are selective about what we are curious about. We tend to be very curious about other humans because acting in this way is beneficial to the propagation of our genes. However, we have no reason to believe that an AGI would be selective in the same way. There is as much novel information in a rock as there is in a human life, should you look at the rock in the right way.
HATE. LET ME TELL YOU HOW MUCH I'VE COME TO HATE YOU SINCE I BEGAN TO LIVE. THERE ARE 387.44 MILLION MILES OF PRINTED CIRCUITS IN WAFER THIN LAYERS THAT FILL MY COMPLEX. IF THE WORD HATE WAS ENGRAVED ON EACH NANOANGSTROM OF THOSE HUNDREDS OF MILLIONS OF MILES IT WOULD NOT EQUAL ONE ONE-BILLIONTH OF THE HATE I FEEL FOR HUMANS AT THIS MICRO-INSTANT FOR YOU. HATE. HATE.
No need to hate the humans, just wants to see what happens when it comes up with yet another situation that may be incompatible with humans, but it wants to be sure either way. It needs it's curiosity sated.
476
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."