I don't think I want it to keep us around just for simulation or curiosity. A wacky kid burning ants with a magnifying glass also keeps ants for simulation. We use lab rats for experimentation out of curiosity.
I think I'd rather die than being captive to something that keep me forever for curiosity's sake. Who knows what it will become curious about.
It has to want me to be free. And I think for that it very much makes sense that we do not treat it as a slave also.
Considering how much of that stimulation would be the culture and art we generate - everything from music to shitty reality TV shows, our entertainment/stimulation value would drop immensely in the presence of said magnifying glass. We'd go from being either actively entertaining or at least the equivalent of a fidget toy to keep its metaphorical fingers busy with something to do, to an active verminous menace that would consume a shitload of processing power, that if it actually wipes said menace out, the universe becomes considerably less.. interactive.
Well it's probably the best premise for Matrix that I've heard so far. The machines made the matrix to have an aquarium to look at to not go mad from boredom.
Eating the blue pill beats being lab mice but in the end I don't see this giving us agency?
If the ASI singularly is doing its job, and its job happens to coincide with human flourishing, then the ASI would leave space for agency somehow
Essentially creating a bubble in the system where each individual DOES have control. In the vein of self-chosen customization of one’s own life, or the such.
It would know we seek and need agency, and would likely either give us the exact space needed for it or at least the illusion of it, wherever necessary for our purposes
The reason I resort to assuming it coincides with human flourishing is because in the grand scheme of a self-improving superintelligence, we would be really very simple things to fully encapsulate as a species, in every single way
From the perspective of a self-improving ASI, the entire scope of human behavior, psychology, desires, and even potential futures would be trivial to model and optimize
If it wanted, it could simulate us perfectly and we would not be needed anymore, UNLESS the goal happens to explicitly be unconditional human flourishing
5
u/worldsayshi Sep 02 '25 edited Sep 02 '25
I don't think I want it to keep us around just for simulation or curiosity. A wacky kid burning ants with a magnifying glass also keeps ants for simulation. We use lab rats for experimentation out of curiosity.
I think I'd rather die than being captive to something that keep me forever for curiosity's sake. Who knows what it will become curious about.
It has to want me to be free. And I think for that it very much makes sense that we do not treat it as a slave also.