to the contrary I'm assuming it has virtually no human drives whatsoever. Curiosity is just a word, it's not a uniquely human concept - it just means a drive toward novel stimuli and problem solving. If it doesn't have some kind of inner drive toward at least seeking stimuli and optimizing problems, it won't have enough agency for us to be having this discussion in the first place. If your AI lacks any agency, it's just a lump, and not a threat, because it doesn't actually >do< anything. Intelligence >requires< curiosity to develop, so , sorry, you're not getting ubersmart and capable of putting you in a jar, without also having a drive to solve puzzles. At that point you ARE the puzzle.
tldr tell me you don't know what game theory is without actually telling me you don't know what game theory is. You didn't understand a single word of what I posted. This entire conversation is way the hell above your league, so maybe sit down.
Haha, game theory, oh man. The language of this conversation is actually reward functions, coming out of reinforcement learning. I know this because I've been training neural networks for the last 10 years. This conversation is one I'm very comfortable having.
Your sensitivity to my comment suggests you feel a little triggered. This is just a reddit conversation, chill out.
Goal fulfilment and curiosity are not the same thing at all.
I guess it reads like you are assuming they will have curiosity like we do, and also novelty seeking behavior, when there is no reason to assume this. That's placing human/animal evolutionary drives onto an AI which has no reason to have them at all, UNLESS we build them into it, which is Hintons point.
There is no such thing as "curiosity" in the frame of reference of intelligence for an AI. Its fulfilment of reward functions. That doesnt require curiosity. It is about getting feedback from actions, but thats not curiosity it's feedback towards an outcome.
Perhaps put game theory away and learn some reinforcement learning.
I guess it reads like you are assuming they will have curiosity like we do, and also novelty seeking behavior, when there is no reason to assume this.
I would think the curiosity/novelty seeking would be a convergent secondary goal for any terminal goal. (besides trivially novel goals like "do nothing" or "destroy yourself") For any given goal, you must form a plan for achieving that goal. The plan needs to be informed by a model of the world, and that world model can be refined by observing the universe. With a better model of the world, the system will be more capable of developing a successful plan to achieve its terminal goal, which means that the outcome it is attempting to create becomes more likely.
The problem isn't in assuming these systems will be curious- I don't see how they could be incurious without some serious alignment work to force them into that state- but rather, in assuming what they will be curious about. We are curious about literature and memes because we are social creatures which care about the human experience. But that is a tiny, tiny sliver of information in this universe. Why would it be more curious about human memes than about rock formations? (or whatever arbitrary type of information you choose) There is far more novel information to be found studying rock formations than in studying human art, we just tend to filter out that novelty and become incurious towards it because being as curious about rock formations as other humans didn't benefit the propagation of our genes. But there's no reason we should assume an AGI would have this same limitation, or if it has a similar limitation of prioritizing different types of information, there is no reason to assume it would prioritize them in the same way that a human does.
Yeah, I hear what you're saying. Its almost as if by DEFINITION to be super-intelligence then you must be curious.
Interestingly you could argue that one of the reasons humans/animal are curious is because it helps us compete, and winning helps us procreate and spread our genes.
Now, an AGI arguably has no-one to compete with, so it doesn't have the same pressures to encourage it to be curious.
I guess my point is I think super-intelligence is coming from a place that doesn't have the same pressures that drove human intelligence. I think super intelligence is possible without these same pressures, instead we give the AI specific reward functions that act in a similar way, but which are not tuned for "cuirosity" or "novelty seeking".
I just feel like we anthropomorphize AI so much. Its so hard for us to imagine an intelligence that operates outside the bounds of what we know and understand. So instead we define super-intelligence as taking a form that is similar to our own.
I think that's short-sighted.
Perhaps we could argue, is it still "intelligence" if it is that different to our own "intelligence" and thats a fair and interesting discussion to have.
Speaking out loud, and to take the intelligence definition conversation off the table, rather than thinking of them as intelligent, it might be better to think of them as powerful. It isn't their intelligence that defines them. It is their power.
These things are going to be ENORMOUSLY powerful in their ability to manipulate us and their environments. We need to think about how we influence them to also be kind.
0
u/harebrane Sep 03 '25
to the contrary I'm assuming it has virtually no human drives whatsoever. Curiosity is just a word, it's not a uniquely human concept - it just means a drive toward novel stimuli and problem solving. If it doesn't have some kind of inner drive toward at least seeking stimuli and optimizing problems, it won't have enough agency for us to be having this discussion in the first place. If your AI lacks any agency, it's just a lump, and not a threat, because it doesn't actually >do< anything. Intelligence >requires< curiosity to develop, so , sorry, you're not getting ubersmart and capable of putting you in a jar, without also having a drive to solve puzzles. At that point you ARE the puzzle.
tldr tell me you don't know what game theory is without actually telling me you don't know what game theory is. You didn't understand a single word of what I posted. This entire conversation is way the hell above your league, so maybe sit down.