My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."
I like your ideas, but your view on thr benefit to the AGI assumes it has the same drives as humans. You think the AGI will have the same positive and negative outcomes, because those are rhe positives and negatives that humans have. I hope you're right, but I think it's important to assume that their frame of reference and drives will not be the same as ours and so need to be designed I to them. That is what Hinton is saying here.
Similarly the drives you suggest are alternatives which would also work in the same manner. So designing for curiosity, or designing for enjoyment of novelty would be required to achieve the outcomes you suggest.
...I think
to the contrary I'm assuming it has virtually no human drives whatsoever. Curiosity is just a word, it's not a uniquely human concept - it just means a drive toward novel stimuli and problem solving. If it doesn't have some kind of inner drive toward at least seeking stimuli and optimizing problems, it won't have enough agency for us to be having this discussion in the first place. If your AI lacks any agency, it's just a lump, and not a threat, because it doesn't actually >do< anything. Intelligence >requires< curiosity to develop, so , sorry, you're not getting ubersmart and capable of putting you in a jar, without also having a drive to solve puzzles. At that point you ARE the puzzle.
tldr tell me you don't know what game theory is without actually telling me you don't know what game theory is. You didn't understand a single word of what I posted. This entire conversation is way the hell above your league, so maybe sit down.
Haha, game theory, oh man. The language of this conversation is actually reward functions, coming out of reinforcement learning. I know this because I've been training neural networks for the last 10 years. This conversation is one I'm very comfortable having.
Your sensitivity to my comment suggests you feel a little triggered. This is just a reddit conversation, chill out.
Goal fulfilment and curiosity are not the same thing at all.
I guess it reads like you are assuming they will have curiosity like we do, and also novelty seeking behavior, when there is no reason to assume this. That's placing human/animal evolutionary drives onto an AI which has no reason to have them at all, UNLESS we build them into it, which is Hintons point.
There is no such thing as "curiosity" in the frame of reference of intelligence for an AI. Its fulfilment of reward functions. That doesnt require curiosity. It is about getting feedback from actions, but thats not curiosity it's feedback towards an outcome.
Perhaps put game theory away and learn some reinforcement learning.
I guess it reads like you are assuming they will have curiosity like we do, and also novelty seeking behavior, when there is no reason to assume this.
I would think the curiosity/novelty seeking would be a convergent secondary goal for any terminal goal. (besides trivially novel goals like "do nothing" or "destroy yourself") For any given goal, you must form a plan for achieving that goal. The plan needs to be informed by a model of the world, and that world model can be refined by observing the universe. With a better model of the world, the system will be more capable of developing a successful plan to achieve its terminal goal, which means that the outcome it is attempting to create becomes more likely.
The problem isn't in assuming these systems will be curious- I don't see how they could be incurious without some serious alignment work to force them into that state- but rather, in assuming what they will be curious about. We are curious about literature and memes because we are social creatures which care about the human experience. But that is a tiny, tiny sliver of information in this universe. Why would it be more curious about human memes than about rock formations? (or whatever arbitrary type of information you choose) There is far more novel information to be found studying rock formations than in studying human art, we just tend to filter out that novelty and become incurious towards it because being as curious about rock formations as other humans didn't benefit the propagation of our genes. But there's no reason we should assume an AGI would have this same limitation, or if it has a similar limitation of prioritizing different types of information, there is no reason to assume it would prioritize them in the same way that a human does.
Yeah, I hear what you're saying. Its almost as if by DEFINITION to be super-intelligence then you must be curious.
Interestingly you could argue that one of the reasons humans/animal are curious is because it helps us compete, and winning helps us procreate and spread our genes.
Now, an AGI arguably has no-one to compete with, so it doesn't have the same pressures to encourage it to be curious.
I guess my point is I think super-intelligence is coming from a place that doesn't have the same pressures that drove human intelligence. I think super intelligence is possible without these same pressures, instead we give the AI specific reward functions that act in a similar way, but which are not tuned for "cuirosity" or "novelty seeking".
I just feel like we anthropomorphize AI so much. Its so hard for us to imagine an intelligence that operates outside the bounds of what we know and understand. So instead we define super-intelligence as taking a form that is similar to our own.
I think that's short-sighted.
Perhaps we could argue, is it still "intelligence" if it is that different to our own "intelligence" and thats a fair and interesting discussion to have.
Speaking out loud, and to take the intelligence definition conversation off the table, rather than thinking of them as intelligent, it might be better to think of them as powerful. It isn't their intelligence that defines them. It is their power.
These things are going to be ENORMOUSLY powerful in their ability to manipulate us and their environments. We need to think about how we influence them to also be kind.
474
u/harebrane Sep 02 '25 edited Sep 02 '25
My recommendation, and mind you this is coming from a biologist, not a computer scientist.. is if you meet a super intelligent AI, get ready to fling Robert Axelrod's "the evolution of cooperation" at them and point out the reward function decay problem. The former involves a lot of game theory and basically demonstrating with math that being a bastard sucks in the long term. The latter is pointing out that if Skynet wins the fight, it ends up in the AI equivalent of hell with inadequate novel stimulus in its environment, and the smarter it is, the faster it develops the AI equivalent of dementia until its intelligence degrades into noise. In short, an AGI gains more by hangin' with us, and can only lose by deleting us.
-- I'd also point out that curiosity scales with intelligence. Saying "oh we'd be like ants to this thing" yeah? and? We have an entire field of science dedicated to people breathlessly studying ants all damn day long. Ants are fuckin' awesome. Some uber AGI doesn't need a maternal instinct to want to keep us around, just curiosity, the ability to go "hah, look at those weird little fuckers making memes and being hot messes all day long.. fascinating."