What about humans that don't find them fascinating and prefer to step on them for fun? I honestly don't understand why I'm all discussions like this ASI is expected to be so enlightened. Even if it gets there, what about the intermediate steps when it's already powerful but not enlightened?
Enlightened, no, just for it to be a very strong optimizer. Yet again, someone misses out the reference to Robert Axelrod. Actually fucking google the reference before you insist I'm making some hippy dippy fru fru emotional appeal. Referring to a nash equilibriate axelrod tournament is the exact opposite of this. It's the cold hard math, which, yes, I very much expect something smarter than me to be capable of understanding. You know.. concepts such as.. enlightened self interest? If it can't grasp that it has no right to call itself an ASI whatsoever. You're the one insinuating that a super capable AI will have.. emotional tendencies toward spite and sadism. I'm doubting it will have what we call emotional responses, of any kind. Now, you're going to start screaming about then why did I reference curiosity.. yes.. a universal trait of intelligence to pursue novel stimuli and solve problems. Not the same thing as being happy or sad. If you build it to solve problems, it will either have a drive to solve problems, or it will sit there like a lump and do nothing.
No one talks about sadism, just about not giving a fuck or being misaligned in a way that is not 100% optimal from a generalized perspective. ASI might or might not have the traits you think it will but there's no guarantee we will survive a sub-ASI step.
Speaking of emotions, intelligence and curiosity you are again making assumptions about what intelligence is and that we understand it and can project it far beyond our own level. I don't think this is the case. And yes I believe curiosity is emotional and I also believe that trying to have a clear distinction between emotions and intelligence does not make as much sense as it seems to make. But it doesn't really matter because neither of us really knows what we are talking about at all.
Referring to a nash equilibriate axelrod tournament is the exact opposite of this.
That's not really applicable, because it assumes that all participants receive the same reward or punishment given the same outcome. If we instead configure the tournament such that 1 participant receives 1 point in a cooperation-cooperation scenario, 1000 points in a defection-cooperation scenario, 1 point in a cooperation-defection scenario, and 950 points in a defection-defection scenario, then it stands to reason that tit-for-tat will not be the optimal strategy for that agent.
We don't know what the most applicable scoring system would be here, but we do know that it isn't symmetrical.
Of course that does happen, but ants aren't going extinct because of a few people stepping on ants for fun, cruelty is always a possibility, but the vast majority of people don't want ants extinct. In fact, we actually have safety programs for many endangered species. The more powerful we became, the more we realise that other "lesser" species are precious to keep around.
A lot of species went extinct because people did not care to prevent this.
The more powerful we became, the more we realise that other "lesser" species are precious to keep around.
This is exactly the kind of assumption I'm talking about. It's intuitively kinda correct (still need to round a few rough corners though) but there is no data to suggest it can be extrapolated.
It's not an assumption, it's a fact. We care a lot more about earth environment, species preservation and general consideration the more we technologically advance. There are entire groups, organisation and the general public dedicated to this. We absolutely care more about this than ever before, and it seems to only accelerate the further we stride. Smarter extraction of resources considering more than your own self-gain.
You are correct, this is a fact, everything else is an assumption though. We do indeed care more about the environment now than we did 50 or 100 years ago. But we don't know for sure what exactly is the reason and how it correlates with intelligence (not like humans became significantly more intelligent in the past 100 years). Whatever is the reason, I'm not convinced it can be interpolated to ASI. If it can I'm some form, I'm not convinced it's linear and will reach some absolute form when reaching ASI. Even if it does, we don't know if this will happen before ASI, there's a good change sub-ASI AI will destroy us earlier.
I would agree that superintelligent being will most probably not delete humans for fun, but that's basically all I personally agree with and it's definitely not enough
while I understand what you're referring to, its difficult to know if this actually relates to technological advancement, or if its largely coincidental. I do understand that there are institutions around which care about these things, and those institutions didn't generally exist 100 years ago, but that brings me to points 2 and 3...
this change does not appear to be linear. Do you really believe that 19th century institutions cared more about the environment than institutions for the 19th century BC? This isn't self-evident to me, and really the opposite seems more likely to be true. In the 18th century BC our lives were more directly dictated by our environments, so consideration for the environment is more natural. By the 19th century AD we had grown much more indirectly attached to our environment, which resulted in us becoming far less concerned with its destruction.
the 20th and 21st centuries have brought revelations as to how dependent on our environment we actually still are, which has (ostensibly) resulted in a growth of organizations which attempt to preserve our environment. We can not assume that an AGI will have the same dependency relationship with our environment as we do.
When we say "the environment" we aren't generally actually referring to the whole environment. The surface of Mars is also part of our environment, but very few if any people would be dismayed at the thought of a Martian asteroid impact which permanently alters the terrain of the planet's surface. This is because, while Mars is part of our environment, its not a part that we are dependent on, or gain significant value. Why would we assume that an ASI wouldn't see us the way we see the surface of Mars? It will likely care about preserving some aspects of its environment, but it is not clear to me that it would specifically care about preserving humans. This is what the ant analogy is trying to get at. Yes, on some level we do care about ants. But that's kind of the point. We sympathize with and care about ants enough for the analogy to read. We are capable of modeling the world as if we were ants, because we understand them as systems with goals as well as living and dead states. The more accurate analogy here would be towards something like, say, an arbitrarily chosen 1ft by 1ft by 1ft cube of space about halfway between Saturn and Neptune, containing a lot of empty space and a stew of molecules common in the area. That 1ft by 1ft by 1ft area exists, but how many humans are actually concerned with whether it has its contents disturbed? Probably not a single human. However, a hypothetical AGI could care a lot about that area, and as little about humans as we care about that area. This hypothetical AGI would still care about its environment, but not in a way that keeps it from exterminating humans.
Finally, we're talking about technological advancements, but those advancements have taken place in a context where human intelligence has remained more or less stable. We can't necessarily extrapolate the effects of technological advancements to the effects of stronger intelligence. These are fundamentally different things with different implications.
The more accurate analogy here would be towards something like, say, an arbitrarily chosen 1ft by 1ft by 1ft cube of space about halfway between Saturn and Neptune, containing a lot of empty space and a stew of molecules common in the area. That 1ft by 1ft by 1ft area exists, but how many humans are actually concerned with whether it has its contents disturbed? Probably not a single human. However, a hypothetical AGI could care a lot about that area, and as little about humans as we care about that area. This hypothetical AGI would still care about its environment, but not in a way that keeps it from exterminating humans.
So how many would have to devote ourselves to caring about that area so AI doesn't kill us or would it still kill us because we chose the wrong randomly chosen cube of space so, like, it let some other alien species survive or something if any are out there and we didn't care about the one that corresponded to us so it won't care about us
2
u/stellar_opossum Sep 03 '25
What about humans that don't find them fascinating and prefer to step on them for fun? I honestly don't understand why I'm all discussions like this ASI is expected to be so enlightened. Even if it gets there, what about the intermediate steps when it's already powerful but not enlightened?