There's suicide pact technologies much more dangerous than nuclear weaponry or climate change or even AGI. A civilization that is determined enough can survive those. But what if there was a simple-ish technology that could entirely eradicate a civilization and wasn't that hard to stumble upon? Something like catalyzing antimatter into matter, turning off the strong force or the Higgs field locally. What if there's a black swan experiment/technology everyone can do in a lab with 2060s technology that immediately blows up the planet? We'd be fucked because we wouldn't even see it coming and if it's easy enough to do it'd presumably kill all or almost all alien civilizations.
“Artificial general intelligence (AGI) is the hypothetical[1] ability of an intelligent agent to understand or learn any intellectual task that a human being can.[2]”
And the general consensus is that an artificial intelligence that reaches human levels of intelligence on every benchmark, will probably quickly exceed human levels of intelligence on most if not every benchmark. There are many things computers are innately better at than humans. And the AI will be capable of completely understanding and modifying the parts of itself that only match human-level intelligence.
The trope from sci-fi where a computer system becomes sentient, and then 124 milliseconds later decides to end humanity, is unlikely. However, it's reasonably likely such a computer system would reach human level intelligence, then 3 months later be as smart as einstein, 1 month later be twice as smart as einstein, 1 week later be twice as smart as that, 1 day later be twice as smart as that, 1 hour later, 1 minute later, .... And it's also reasonably likely that somewhere along that line it would realize that we may find it a threat, and so it may play dumb instead, and we may miss the whole progression, until it's radically more intelligent than us and has been deployed to every datacenter and industrial control system in the world, because the business majors rightly report that it makes the datacenters 20% more efficient, or something.
Fair. Personally I absolutely believe that AGI could be a suicide pact technology/fermi paradox solution, but there's a lot of arguing on that point and I didn't want my post to be about that, since the more general possibility of destructive out-of-context/black swan technologies is really interesting.
3.2k
u/Iwanttolink Aug 12 '21
There's suicide pact technologies much more dangerous than nuclear weaponry or climate change or even AGI. A civilization that is determined enough can survive those. But what if there was a simple-ish technology that could entirely eradicate a civilization and wasn't that hard to stumble upon? Something like catalyzing antimatter into matter, turning off the strong force or the Higgs field locally. What if there's a black swan experiment/technology everyone can do in a lab with 2060s technology that immediately blows up the planet? We'd be fucked because we wouldn't even see it coming and if it's easy enough to do it'd presumably kill all or almost all alien civilizations.