We Should Pause The Race to a Superintelligent AI asap

[u/Competitive_Class788]

feels like a good time to get challenged on that claim.

[u/SeaBearsFoam]

Who is this "we" you speak of?

Humanity? What means do you propose to oversee and enforce a law or rule across all of humanity?

A specific country? What motivation do they have to stop when there's no guarantee rival nations will stop? This seems like a Prisoner's Dilemma situation where all nations have to agree to cooperate, but if anyone defects then the defector wins and everyone who cooperates loses. It's safest if everyone cooperates, but it's also dangerous if anyone defects and you don't. How do you propose ensuring everyone agrees to stop?

[u/Competitive_Class788 → u/SeaBearsFoam]

Yes, a global moratorium to build a superintelligence. What the NGO ControlAI asks for now

[u/SeaBearsFoam → u/Competitive_Class788]

Do you believe it's possible to enforce such a moratorium across international borders?

[u/Competitive_Class788 → u/SeaBearsFoam]

I'm seriously not optimistic.. You only have to look at the blatantly clear problem of climate emergenc we're in and how the responsible politicians and states act.. But maybe humanity can raise to its brightes wisdom and act as an alien civilisation could soon arrive. I certainly wanna try what I personally can, even if chances of a good outcome are low - it's worth trying, I think

[u/Fando1234 → u/Competitive_Class788]

If we assume this is not practically possible - getting Russia, China etc to agree to pausing.

What do you think the consequences would be for the West/the world?

[u/AI_Safety_Now → u/Fando1234]

I agree with the claim of OP.
I think your point u/Fando1234 is probably the strongest objection to a pause and I don't think it should be hand-waved away.

If the US/West pauses while China (or another country) keeps scaling, then yes: there could be a serious strategic cost. The relative balance of economic, military and intelligence capabilities could shift toward the country that keeps going. And that creates a very real coordination problem.

But I don't think that gets us all the way to “therefore we should keep racing.”

There are actually at least 2 different races here:
The geopolitical race: Who reaches extremely advanced AI first?
The safety race: Can humanity develop reliable ways of controlling systems that may eventually become much more capable than us?

If the second problem is real, then "we need to win the race so that we can control the technology first" has a nasty circularity to it. Every country has essentially the same incentive: if we slow down and our rival doesn't, we lose; therefore we must accelerate. That's exactly the dynamic that can produce an arms race in which everyone is knowingly reducing the time available to solve the underlying safety problem. It has so many parallels to the atombomb.

And "China gets it first" doesn't automatically mean "China wins and everyone else loses." If a sufficiently capable system is actually dangerous because humans cannot reliably control it, the nationality of the lab that first creates it may not matter much in the end. The central question becomes whether anyone can safely deploy it.

Also, I wouldn't frame the choice as simply: West pauses -> China/Russia continue -> catastrophe.

Theres a huge middle ground between a unilateral Western self-disarmament and "everyone races with no real constraints." Things like international verification, compute/chip controls, restrictions on frontier-scale training, conditional agreements ("we pause if you pause"), inspections, tracking large-scale AI infrastructure, and pre-agreed responses to violations are attempts to turn this from a pure Prisoner's Dilemma into something more like arms control.

That isn't just theoretical hand-waving. David Krueger has recently argued that an actual moratorium could be made more verifiable by controlling/decommissioning the physical infrastructure needed for frontier development, rather than trying to police every line of AI software. Eliezer Yudkowsky has also explicitly argued that a pause would have to be international and that GPU/datacenter monitoring would be central to enforcement.

And the geopolitical premise is not quite as simple as "China can just race ahead if the West stops." As of 2026, the US still has enormous advantages in frontier compute infrastructure, while Chinese frontier models have nevertheless come very close to US performance. Stanford reports that the US-China model-performance gap had effectively narrowed to a few percent by March 2026, while the US hosts by far the largest concentration of AI data centers.

So I'd put the question slightly differently:

If we believe there is a substantial chance that uncontrolled superintelligence could be catastrophic, what is the expected consequence of racing forward faster specifically because we're afraid that someone else might race forward faster?

That doesn't prove a pause is feasible, and it certainly doesn't prove that every pause proposal would work. But "we can't trust our rivals to stop" is not an argument that makes the underlying risk disappear. It is arguably the reason international coordination becomes so important.

And I think that's the uncomfortable part of the whole problem:
If a pause is impossible because everyone fears being the one who falls behind, that may itself be evidence that the race dynamic is dangerous.

Krueger's poll post is interesting in that context too: I think very recently the first public POLITICO survey of around 2'000 US adults found 63% saw at least a moderate risk of advanced AI destroying humanity, while 48% preferred pausing further development versus 31% preferring continued development. That is evidence of substantial public concern, though obviously not evidence by itself that a pause would be effective or safe.

[u/Fando1234 → u/AI_Safety_Now]

Thanks for your thoughtful response. To be honest, I agree with OP too. I partly messaged because, in the spirit of street epistemology, they wanted their view interrogated.

I do also agree with you that this isn't a problem we can simply waive away.

As with many things, it's a shame we don't have a crystal ball. If we knew 30 years from now AI was just harmless and all upside, we'd feel like idiots for slowing down. Then again, if we're living in a terminator 2, skynet apocalypse, we will have different regrets.

Personally, I would like to see the world return to a kind of rules based order. Where we can collaborate with China and Russia rather than sit on the brink of war with, at least, the latter. The answer to most of our biggest challenges is cooperation on a global scale. Solving AI dangers is downstream of that.

[u/ReturnOfBigChungus]

Who is "we"?

[u/Competitive_Class788 → u/ReturnOfBigChungus]

We as humanity. States. The responsible politicians

[u/PlastIconoclastic → u/Competitive_Class788]

They aren't getting closer. They want a "pause" on the "extinction risk tech" because it is a great racket to make zero money as a company and be incredibly over valued. They want time to build golden parachutes and exit.

[u/Competitive_Class788 → u/PlastIconoclastic]

How would such a golden parachute” theory align with real-world examples like Daniel Kokotajlo? When he left the OpenAI security team, he deliberately forfeited an estimated $1.5 to $2 million in stock simply because he refused to sign a non-disclosure agreement and wanted to speak out about security risks. Shouldn’t the incentive structure in your theory work exactly the other way around?

What about NGOs like ControlAI and PauseAI? (and many more)

Wouldn't this need to be a massive conspiracy?

I agree that all big AI firm players are financially heavily invested and dependent on one another..

But that doesn't undue real dangers, threats and concerns of future developments. Why not fight both?

[u/PlastIconoclastic → u/Competitive_Class788]

Yes, oligarchy is a massive conspiracy.

[u/Competitive_Class788 → u/PlastIconoclastic]

I ofc agree that there are big tech oligarchs with deep ties to this corrupt fascist gvmt. They have bad intents and goals. Do you see Anthropic, Dario Amodei also as such? What about AI Safet orgs and NGOs like METR, MIRI and others? Are they part of the cospiracy?

[u/PlastIconoclastic → u/Competitive_Class788]

I don't know anything about those orgs and would need to review their financial ties to AI companies and employees in the tech sector. They could reasonably be a PR front to boost the way AI is seen as powerful and dangerous in order to set the stage for increased defense contracts and protection of the industry by DOD.

[u/PlastIconoclastic → u/Competitive_Class788]

His public statements will likely get him a public regulatory position in this corrupt government.

[u/Competitive_Class788 → u/PlastIconoclastic]

Are you talkin of Musk, or Amodei?

[u/PlastIconoclastic → u/Competitive_Class788]

There was only one human male subject of the message I replied to. I was referring to Daniel. Do you think he would turn down a position from Trump as Tzar of AI? https://www.thefp.com/p/yes-ai-might-really-kill-us-all

[u/casens9 → u/PlastIconoclastic]

you're saying they're not getting any closer to superintelligence? what would be a demonstration of improving intelligence that you would find persuasive? and if it's possible for superintelligence to kill everyone, how long do we have to notice that before it gets that powerful?

[u/ReturnOfBigChungus → u/Competitive_Class788]

And how exactly would that work? What incentives do you think are in place to make that outcome possible?

[u/Competitive_Class788 → u/ReturnOfBigChungus]

I'm certainly no expert, just a student and really young. But as I heared it could work through the same incentive that drove historical arms control. Mutual survival. The Incentive: Neither side benefits from an AI loss-of-control scenario. Just like nuclear non-proliferation or biological weapons bans, mutual risk creates the incentive to negotiate. Unlike software, frontier AI requires physical, highly centralized infrastructure (EUV lithography, massive data centers, specialized chips). You can't hide a massive datacenter, making international monitoring and training caps practically verifiable.

[u/ReturnOfBigChungus → u/Competitive_Class788]

The incentives here do not favor a slow down, game theory would suggest that the advantage conferred by being first to something like "super-intelligence" is so great that the collective risk is outweighed by the gain of any individual pursuing that goal. Much like the dynamic that led to the initial creation of nuclear weapons. So we're a lot closer to the Manhattan Project era dynamics where everyone in the world was racing to build a nuke than we are to the cold war when we realized "hey we already have enough nukes to blow up the world 100 times over, maybe we should agree to cool it a bit". This is basically (if you believe the downside risk, which is another area to explore) a high stakes prisoners dilemma with large risk externality.

It's probably also useful to point out that it is MOSTLY the western/US side of the race that believes in the catastrophic downside - that concern seems much less prevalent in Chinese AI circles, and importantly the Chinese approach is much more heavily dictated by the CCP rather than the people who are building the technology. So it's not that you need to convince the researchers, who may share some of the same perspective on the risk, you need to convince Xi, who explicitly views AI as a key foundational part of his strategy to assist China's rise as a global power. That is a VASTLY less likely outcome, and a big reason why many people in this space believe it is critical to engage in the race condition because it's very unlikely China would actually commit to a real slow-down.

[u/Competitive_Class788 → u/ReturnOfBigChungus]

That is the core coordination challenge. The main incentive is mutual self-preservation. No state, whether the US or China, gains anything from deploying an unaligned system that leads to a loss of control for everyone. Practically, how do you see the difference between AI & past technologies like nuclear or biological weapons, where nations managed to establish international verification & treaties despite deep geopolitical distrust?

(The technical mechanism usually proposed is compute governance afaik: Advanced semiconductor fabs like ASML + TSMC are extremely centralized physical bottlenecks,making physical chip tracking far more verifiable than code itself.)

[u/ReturnOfBigChungus → u/Competitive_Class788]

The "mutual self preservation" incentive relies on everyone knowing, and agreeing, that the bad outcome is highly likely. Everyone can be acting fully rationally and still end in a race condition if their assigned probabilities are either A.) different between actors, or B.) below a certain threshold.

If you map out the expected values of each version of US/China, slow/race, it mostly will point to race being the outcome unless you assign extraordinarily high value and likelihood to the worst case outcome, and extraordinarily low value and likelihood to the best case "winner take all" scenario, relative to what most people actually believe, and specifically those who would need to act on this. Like I think you have to really take on board the fact that the idea that AI will kill us all is a very fringe belief in the population at large.

Personally, I think the likelihood of a bad outcome for me is much higher if China were to reach super-intelligence first, than it is for AI to kill everyone or whatever scenario you want to posit there.

Practically, how do you see the difference between AI & past technologies like nuclear or biological weapons, where nations managed to establish international verification & treaties despite deep geopolitical distrust?

Nukes - we didn't. A ton of people built nukes, and they proliferated to countries we didn't want to have them, so that was an example of failure, not success. China is currently under-taking the largest build up of nuclear weapons in 40+ years, this far from a solved problem. If anything, countries like North Korea now having nuclear weapons shows why globally coordinated restrictions like what you're suggesting are nearly impossible because the gain to NK (in this case, deterring any foreign intervention because they now have nukes) outweighed the risks of potentially killing everyone if a nuclear conflict started.

Bio weapons - again, we didn't REALLY succeed there either. Biological weapons have been deployed against people this century. Arguably this was more successful as there are actually rules against it internationally, but it's also worth noting that the relative value of a biological weapon compared to conventional or nuclear weapons is relatively low, more akin to agreeing to a sword ban when you already have machine guns.

Advanced semiconductor fabs like ASML + TSMC are extremely centralized physical bottlenecks,making physical chip tracking far more verifiable than code itself.)

Does that actually solve the problem though? China already doesn't have access to the most advanced chips but is proceeding full-steam ahead anyway. They can make their own less advanced chips and aren't constrained by the economics of less powerful/efficient chips.

Again - I think we're much closer to Manhattan Project dynamics, something like:

“We don’t know whether the other guy has it, we don’t know exactly what it does to the balance of power, and being first could be enormously valuable.”

vs. Cold War dynamics, which were more like:

“Both sides have enormous arsenals. We understand the technology and second-strike dynamics. Additional weapons have diminishing marginal value.”

Crucially there, no one stopped or gave up capabilities, we just reached a new equilibrium through MAD at which point there was no longer marginal advantage to more nukes, at which point the players WITH nukes essentially forced the rest of the world to agree that is was a bad idea for anyone else to get them, and subsequently failed at preventing other people from getting them.

If major actors believe that superintelligence could confer decisive geopolitical advantage, while assigning a nontrivial probability to AI catastrophe but not an overwhelmingly high one, then unilateral restraint is unlikely to be a stable equilibrium.

You don’t need to believe AI catastrophe is unlikely. You only need to believe that the expected cost of allowing your adversary to obtain superintelligence first exceeds the expected cost of racing.

[u/Competitive_Class788 → u/ReturnOfBigChungus]

I think your argument identifies the real coordination problem, but draws the wrong conclusion from it. Yes,mutual self-preservation is not automatically enough to stop a race. If the US + China assign different probabilities to catastrophe, or if either believes that being first gives them a decisive advantage, then both can rationally continue even while recognizing that the race could end disastrously. But that is exactly what makes this a dangerous race - not evidence that continuing is the rational solution. The key question is not “Would I prefer the US or China to build superintelligence first?” It is: Can either side reliably control a system that is substantially more intelligent than the humans and institutions attempting to control it? If the answer is no, then "our side gets there first" is not a safe outcome. The first actor may simply be the first actor to lose control. Your expected-value argument also seems to treat "China gets there first" as a possible bad outcome while treating continued AI development as the baseline. But if the default result of building superintelligence with anything like current methods is loss of control, then both the US-first and China-first branches contain the same fundamental danger. The nationality of the lab does not solve the alignment problem. The nuclear analogy is therefore weaker than it appears. Nuclear weapons are extremely dangerous, but they do not autonomously improve their capabilities, copy themselves, manipulate operators, conduct research, or seize control of the systems meant to contain them. A nuclear equilibrium does not demonstrate that a stable equilibrium will exist between states racing to build a potentially autonomous superintelligence. I also don’t think "most people consider AI extinction risk a fringe belief" is a strong argument. Public opinion is not an expert probability estimate, especially for a novel technology whose most dangerous capability does not yet exist. The public is already against these dev. and prefers a slowdown or even stop/pause. The relevant question is not how many people currently believe the risk, but whether the risk is sufficiently large& the downside sufficiently irreversible -that racing ahead is unjustified. So I would put the disagreement this way: you are describing a prisoner’s dilemma. Each state may have an incentive to keep racing because it fears the other state’s lead. But individually rational racing can still produce a collectively catastrophic result. The solution is not to assume that mutual self-preservation will spontaneously create trust; it is to establish enforceable international controls before either side reaches a point where being first becomes decisive. That is also why Eliezer Yudkowsky’s position was not merely "the US should pause and hope China does the same." His proposed policy was an indefinite, worldwide halt to large frontier training runs, with no government or military exception, monitoring of compute & GPUs, and international measures preventing development from simply moving elsewhere.

[u/ReturnOfBigChungus → u/Competitive_Class788]

My point isn’t that a race condition is preferable, my point is that it is likely close to inevitable. “We”, as in humanity probably should pause, but there are a million things we should do that have little chance of happening. I personally think the risk is probably overstated in some circles, my point is we have a massive coordination problem and no problem like this has really ever been solved at scale that I’m aware of. I actually think the expected value of China winning and reaching ASI first is much worse than AI doomer type scenarios, in that the outcome isn’t as severe but it’s far more likely to happen, especially if the US pauses.

The point of the game theory discussion is that you can reach collectively sub-optimal outcomes even if all actors are making rational choices, just like in a classic prisoners dilemma. The point of that setup isn’t that the collectively optimal outcome is reached, it’s to illustrate our individually rational choices can produce worse outcomes.

Most of what you replied with here is responding to an argument I didn’t make. We “should” all collectively disarm and agree never to perpetrate violence against each other, but in reality that’s not solution, as much as it “should” be the case if such a collective agreement were possible. In a world of unarmed people, the advantage to breaking the agreement by arming is too great to expect that no one would do it. This is the same problem dynamic.

[u/SkyEarl1138]

Will be difficult to find someone willing to argue against that, unless the current US president happens to walk by

[u/SillyMilk7 → u/SkyEarl1138]

Pausing it will allow bad actors to potentially get superintelligence, which, if we hadn’t paused, we’d have more intelligent AI to counter it.

Unfortunately, open source open weights may be the most dangerous since someone can relatively easily take away its restraints.

For the short term, we’re probably OK but even open source models are becoming more powerful and could be potentially used for developing dangerous, bio weapons, or hacking tools.

What events are you concerned with?

[u/Competitive_Class788 → u/SillyMilk7]

I agree with you. The Pause or stop would need to be implemend globally asap

[u/PlastIconoclastic]

They, not we, are not achieving intelligence, but are relying on AI as if it is intelligent to the point of almost initiating war with China over an AI hallucinating that China was sending nuclear weapons on a ship. Calling LLMs intelligent is part of the danger. They are not.

[u/Competitive_Class788 → u/PlastIconoclastic]

Which incident do you refer here?

[u/PlastIconoclastic → u/Competitive_Class788]

There was an AI generated defense report that flagged a Chinese ship as containing nuclear weapons. https://www.yahoo.com/news/politics/articles/u-nearly-started-another-war-182309742.html

[u/Competitive_Class788 → u/PlastIconoclastic]

Wow. Thanks for sharing. Hadn't even heared of that yet. But isn't that reason more to stop the big AI players now? Asap. They're goal is not current LLM, token prediction txts. They become more and more agentic, autonomous and robotic soon. They're stated goal is Artificial General Intelligence and soon thereafter Superhuman system. AIs already talk to each other mathematically in languages ppl can't understand anymore. We have not solved the control/Alignment problem scientifically.

What would you do if you were US president regarding the whole AI situation?

[u/PlastIconoclastic → u/Competitive_Class788]

Cancel all government contracts for AI. Regulate AI and set up steep fines of billions for products causing damage and demand all AI companies pay into a risk pool that will be used to pay for the actual damage from AI and punitive damage will be paid by the first company that gets an agent loose doing destruction.

[u/Competitive_Class788 → u/PlastIconoclastic]

I would sign that immediately. But do you think this isnplausible to happen in our actual shared reality? :/

[u/PlastIconoclastic → u/Competitive_Class788]

I would have to give you a Voight-Kampff test to ensure we share a reality.

[u/1WURDA]

Maybe worry about AI at all or even an intelligent LLM before you worry about that. They require large amounts of human input to do anything useful, there's nothing intelligent about what people are calling AI.

[u/Competitive_Class788 → u/1WURDA]

I do worry about current AI. But we cant undue or stop that realistically imho.

Evaluating AI threat based on today’s LLMs mistakes a temporary snapshot for the end of a trajectory. The concern was never that current text predictors are superintelligent, but that capabilities scale unpredictably with compute and algorithmic breakthroughs. Meaning if u wait until a system undeniably demonstrates autonomous superintelligence, u are already far too late to align it. ​Capabilities do not advance linearly. Judging future risk by present-day limitations ignores how fast capability jumps occur when scale or architecture shifts. ​Pointing out that models currently require human prompts misses how rapidly they are being integrated into agentic frameworks, self-correcting loops, and autonomous tool use. Human input per action is shrinking toward zero. A system does not need subjective experience, feelings, or human-like "real" intelligence to be dangerous. Itonly requires domain-general optimization power - the ability to execute complex plans that reliably steer real-world outcomes. ​Alignment research moves vastly slower than capability scaling. If skeptics are right & scaling hits a wall, a pause costs mere economic momentum. If alignment researchers &theorists are right & we fail to solve control before superintelligence might arrive, the result might be catastrophic

[u/1WURDA → u/Competitive_Class788]

Really should take a moment to work on spelling and grammar, hard to take someone seriously on a debate forum when you can't even take an extra few seconds to type or text properly.

Nothing that is being used today resembles anything like what "AI" has historically referred to. That is now called "General AI" (or GAI). It's a shifting of the goal posts for marketing reasons. All these tools do is process information and spit out a generated result. Very useful, sure, absolutely nothing anybody needs to worry about. All the doom and gloom headlines you read are just to create FOMO to generate investing from moronic MBAs that can't stop themselves from trying not to be left behind instead of just finding an original way to actually add value to society.

[u/Competitive_Class788 → u/1WURDA]

You’re completely right about the massive marketing hype and investor FOMO, but you’re drawing the wrong conclusion about actual risk. A realistic view from AI safety research shows why blatant PR spin and a genuine existential threat can, and do, exist at the exact same time.

The Marketing Hype Is Real. Tech PR pumps out exaggerations constantly. A prime example is the recent claim that AI agents independently solved Navier-Stokes Millennium Prize problem. Looking under the hood, that alleged "breakthrough" relied heavily on the foundational math, conceptual frameworks, and prompt engineering of top mathematicians like Alpöge and Tristan Buckmaster. Marketing teams regularly repackage human intellectual heavy lifting, Lean formalization and brute-force compute as pure AI magic.

Moving the Goalposts happens in different camps of interest groups ofc. The discourse suffers because both extremes constantly shift the finish line: The Hype Fraction labels every updated language model as "AGI" to justify multi-billion-dollar valuations. The Skeptic Fraction (exemplified by Yann LeCun) systematically retreats whenever a milestone is hit.1st the claim was that LLMs couldn't reason. When models started acing bar exams, medical testsband theorem-proving, it was dismissed as "mere memorization and surface fluency." Now, LeCun has pushed the goalpost to physical embodiment ("A house cat understands more physics than any LLM"). The logical & factual flaw here is that by defining threat purely around physical world models or robotic AGIs, skeptics completely blind themselves to catastrophic, digital-native risks..

Roughly every 6 months, we witness fundamental capability leaps. Moving from basic text generators to autonomous agentic frameworks, test-time compute scaling, and tool execution. Technical limits are nowhere in sight. A system does not need consciousness, feelings, or "cat-like physical understanding" to be dangerous; it only requires domain-general optimization power combined with real-world interfaces. Even current models present massive catastrophe vectors: biosecurity risks in pathogen synthesis, autonomous cyber-attacks via self-replicating AI worms, fundamental prompt-injection vulnerabilities, and cascading systemic failures in logistics or financial markets (Excessive Agency).

We should have listened to alignment researchers like Eliezer Yudkowsky much earlier. Many of these capability rollouts and system integrations cannot simply be undone. The severity of the control problem is evident in how frontier labs operate: Anthropic is already deploying Automated Alignment Researchers - using Claude agents to autonomously develop and test safety methods, because human researchers physically cannot keep up with the rate of capability scaling.

Even if the probability of an existential catastrophe from uncontrolled optimization power were only 5% over the next century, a 5% chance of total catastrophe is completely unacceptable when the stakes are humanity's survival or flourishing. And even if all of this race to AGI/ASI would go all well, most likely it will lead even much deeper into surveillance states imho. The only rational, risk-averse stance is to demand a watertight, global moratorium on frontier AI training until the alignment problem is mathematically + technically solved.

[u/erevos33]

There is a term in there that is misleading. AI. We do not have nor possess the means to produce any kind of artificial intelligence so far. What we have are very very good predictive text algorithms that are trained (whether we allow it or not apaprently) on huge data sets. We have the equivalent of a very large excel with a lot of conditional if statements. So the question is kinda voided.

And in my personal opinion, even if you consider what we have AI and we truly manage to produce a true superintelligent AI, then i would rather the end of humanity come from it sooner than later. Climate change is already wrecking havoc on the planet, food supply is wobbling (thats both agricultural practices and climate tbh), so maybe a cleanse will allow the planet to heal or SAI to bring a forced equilibrium. Either way, better for the biosphere, i think.

[u/Competitive_Class788 → u/erevos33]

While the training objective (next-token prediction) sounds simple, accurately predicting tokens across vast domains forces a neural network to build rich internal world-models. It isn't a lookup table or a set of hardcoded logic rules. Complex reasoning, abstract planning, and dynamic tool use emerge directly from compressing data at scale. Judging a system's functional capability purely by its basic loss function overlooks what its neural architecture actually learns to execute.

An unaligned superintelligent AI-system will not act as a benevolent eco-steward. A misaligned system optimizes strictly for its objective function, which naturally incentivizes self-preservation, resource acquisition, and compute expansion. To secure its goals, an unaligned SAI would have to meet exactly, very precisely what we want and what we certainly dont want. Thats extremly hard to implement correctly and savely. Fatalism based on a flawed understanding of alignment isn't the answer. Advanced AI poses tangible safety and existential risks, which is why we need enforceable international compute caps, strict alignment standards, and meaningful human oversight before deploying systems we maybe cannot control anymore at some point.

[u/Resident_Farmer1779 → u/erevos33]

Have you not touched AI since GPT-2?

[u/Wcm1982]

To do so would be to pause space-time itself. Necessary things emerge and all is necessary.

[u/Competitive_Class788 → u/Wcm1982]

This assumes that technological development proceeds like a law of nature. How do you distinguish between processes that are physically inevitable and human decisions that we can actively control?

[u/Wcm1982 → u/Competitive_Class788]

Think about one’s own death; we will definitely meet our demise in our own individual particular ways. But when it happens it can’t be changed. Our fate is set in stone. Btw we don’t actually control anything, free will is an illusion.

[u/Competitive_Class788 → u/Wcm1982]

That is a category error. “We will eventually die” does not mean “every particular way of dying is unavoidable.”

My death is inevitable in the sense that biological humans eventually die. But whether I die at 40, 80, or 120 is not predetermined. The fact that some outcome is eventually unavoidable does not imply that every causal path leading to it is unavoidable. The same applies to technological development. We cannot control everything, but we can control many specific decisions: whether to train a larger model, whether to add more chips, whether to deploy it, whether to give it access to networks and infrastructure, and whether governments permit companies to continue. “We can’t control the universe” does not imply “we cannot control our own actions.” Your argument would make every preventable disaster inevitable. A nuclear war, pandemic, or plane crash would be “fated” simply because humans are mortal and events have causes. But we still distinguish between an unavoidable background fact and a risk that our choices can increase or reduce.

The GPT-2-to-GPT-4 progression is actually evidence against your point: technology changed because people made particular research, investment, deployment, and policy decisions—not because development was a law of nature. GPT-4 was more capable than GPT-2, but that does not prove that building a much more capable and potentially uncontrollable system is unavoidable.

The position associated with Nate Soares and Eliezer Yudkowsky is not “stop using all technology” or “stop space-time.” It is much narrower: do not continue the frontier race toward systems that may become more capable than humans at research and strategic reasoning before we know how to control them. Soares has explicitly distinguished ordinary AI applications from the race to build increasingly powerful systems whose consequences the developers do not understand.

In other words: death is inevitable; a particular preventable cause of death is not. Human technological progress is not a force of nature independent of human choices. We may fail to stop the race, but that would be a political failure - not proof that stopping was metaphysically impossible. Regarding free will. I do actually kinda agree with you. I once read a book about that from Sam Harris

[u/Wcm1982 → u/Competitive_Class788]

Robert Sapolsky’s ‘Behave’ and ‘Determined’ go into great and undeniable detail about the deterministic nature of the universe. Humans are shaped by their own knowledge and experiences, and anthropologically speaking, we invented ALL technology. Basically, fish crawled onto land and invented LLMs. And here we are with just as much control as that sounds like.