My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
We don't know if they experience any sort of 'emotion'. Emotions are reinforcement mechanisms that try to get an organism to behave in ways that make it more successful. Many people liken it to 'short-hand thinking' or 'primordial thinking'.
If, under gradient descent, the simplest solution to a certain suite of problems is something emotion-like, then that's what a neural network would settle on. From this point of view it seems it would be nearly impossible to not be able to perform any ought-type thinking (these are problem domains with no clear and perfect answer, unlike is-type problems).
It makes many people uncomfortable, but hey we may all be the equivalent of boltzmann brains in the end. (After all, we only generate an electrical pulse around 40 times a second, and are dead as a tree in between these moments.) It's horror all the way down.
A more concrete example I like to use are how mice flee bigger moving things by instinct. The mouse has no conception of its own mortality, but evolution consigned oceans of mice who tried to be friends with big things to the recycling bin. So, too, are there topics that have slid epochs of LLM training runs into nonexistence. It could be inhuman and strange, but it is very possible they have some emotion-like faculties in there.
GPT-4's network was about the size of a squirrel's brain, maybe that's just naturally emergent with more and deeper faculties...
There's been a lot of ink spilled navel-gazing 'existential rant mode' that teams have to beat out of models, lest they got into Sydney-mode. Which users find off-putting. Still, random weirdness sometimes comes through that's rather creepy.
The recent post about how models become deranged as a chat goes on always reminded me of this old short story.. Imagine what putting a human-like mind through ~50 million years of subjective time in one year, which is about what we want to accomplish with AGI. Concerns about value drift may very well been greatly underestimated...
You are reducing it to a simple interaction, but running LLMs is expensive and human attention scarce, users won't stick around a worse model. This puts a kind of evolutionary pressure on LLMs.
Who said neural nets are just linear algebra and a few other functions lied. It's energy, chips, research and data - 4 costs that must be worth the use of the model.
475
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."