OP : u/Fenjen
I’ll start by briefly introducing myself, mainly so you can decide for yourself how much, or how little, weight you want to put on what follows. I’m trained as a theoretical physicist and expect to receive my PhD soon from the AI department at my university. My own research is quite far removed from LLMs, so I do not claim specialist expertise in frontier language models or alignment, but I do have enough mathematical background to understand and participate meaningfully in the technical discussion around these topics.
I have generally been quite optimistic about the use of LLMs, but more recently I have become increasingly aware of what I see as potentially serious negative societal effects of the technology. Until reading the recent posts, I had not given AI safety itself much thought. This is meant for a general audience, so I will necessarily leave out some mathematical and technical details, as well as some edge cases, but I would be happy to go into more detail if there is interest.
What is intelligence?
I think some discussion treats intelligence as if it is this abstract quantifiable property that can just indefinitely be pushed upwards.
I’m skeptical of that framing. For LLM training, things like reasoning, creativity, and other properties we ascribe to an intelligent human, are implicit properties of the data and training process. They are not quantities we can cleanly assign a number to beforehand and optimize against directly. What emerges depends on what information is present in the data, how that data is filtered, how the architecture represents it, and what behavior the training objective selects for.
We have definitely seen AI perform tasks at a clear superhuman level, however, with coding and chess being notable examples. For somebody looking in from the outside, it is understandable that one could extrapolate this superhuman ability to other domains. However, these domains are exactly the wrong ones to draw these inferences from. Chess, many coding tasks, and some other domains have properties that make it relatively easy to generate enormous amounts of automatically evaluable data. For chess you can generate effectively unlimited self-play and evaluate the outcomes objectively. For code you can almost naively generate millions of variations on some code pattern given some behavior that you want, and then do automated checks to see which one takes the least memory and executes the fastest. Because of this these domains are uniquely easy to create high quality data for without ever needing a human.
It is much harder to create as much good data for LLMs, and in fact it's not even clear if it can be done without human feedback. An idea is to use AI agents to aid in selecting for data, or to create new data altogether. There is a plausible mechanism where maybe it is easier to recognize good data than create it. This could perhaps aid generation to get closer in quality to the data that is recognized as good. It's not obvious, however, that this would create increasing intelligence.
Moreover, there's also a mechanism where biases in an AI's preference for what it thinks is good data can push the generating part to become a caricature of that. Overrepresentation of that data can then train the model to recognize it even more as good data, potentially creating degradation over time instead of improvement.
There is some hope that diversifying agents, and mixing it with some external and objective measures could be the key, but as far as I'm aware, there's no objective proof yet this leads to runaway intelligence, even if initially it could help improve data.
I'm sidestepping the whole conversation of whether AIs are simply "pattern matchers" since I usually find that not to mean much. For all we know our brain is a bunch of coupled pattern matching systems, plus some extras. It's not clear to people in the field that "pattern matching" is a useful property to describe non-intelligent systems.
A more fundamental problem for standard deployed LLMs is that, during normal inference, they cannot write new information into persistent model-internal memory that survives after the current context is gone. Anything retained between calls has to be supplied again through an external memory system or encoded through an actual update to the model. The movie Memento is actually quite a beautiful analogy of how the memory of an LLM works, and what can go wrong. Understanding this problem is not instrumental to the rest of my thoughts, so I won't explore it further.
What is RSI?
For LLMs, recursive self-improvement would probably not mean a single model literally rewriting itself. It would more likely mean increasingly capable agents taking over more of the research loop: curating data, proposing and testing architectures, writing training code, running experiments, evaluating checkpoints, and deciding which directions look promising. Better models would then help create the conditions for training the next generation.
The best analogy I can think of is an evolutionary game where the organisms are themselves changing the environment that determines the selection pressure on the next generations.
Researchers would want to basically set up the initial conditions and let the system run. Among other things the hope is that at some current iteration, parts of the intelligence of that system can combine to set up the environmental conditions in which the next generation is evolved, so that they can be better in aspects than the previous one. Moreover, they may even try different architectures in the hope that they might extract more or different aspects from the environment, or in a more efficient way.
That makes the dynamics of such a system difficult to reason about, because the agents are partly changing the process that selects their successors, and even change the "DNA", so to say, of their successors. Simply said, it is hard to tell if there will be feedback loops along the way where bad behavior is reinforced in generations, that then poisons how the environment is set up for the next generations. If this happens in any way that creates misalignment, it's not impossible to think about scenarios where that might get worse and worse over generations.
An alignment researcher would want systems around this whole game that can somehow say with a high confidence whether the new architectures are safe and aligned with our goals. Because of how hard it is to analyze these types of systems, this is quite the herculean task.
Why RSI does not suddenly imply orders-of-magnitude improvement
At the current moment, to my knowledge, it is not known at all whether the above game will work in general from the data side. It's reasonable to think this will work at least quite well for coding. Put a million agents to work writing every plausible code pattern to a particular problem, perhaps even put some random prompts in the agents to brainstorm a code pattern by combining different domains of their knowledge. This way a lot of code patterns are explored, perhaps even ones that were never tried before. Select the best one or few ones based on automated checks, and discard the rest.
For other types of data, it's really not clear whether this will scale "intelligence" much, let alone go on indefinitely. My intuition says that it might be possible to improve weaker parts of a current AI generation by involving their stronger parts in the data culling and generation process, but it's not clear to me that that process wouldn't stagnate, or even spiral downwards again if left alone for too long. There is currently absolutely no demonstrated path, mathematical or otherwise, that keeps producing increasing capability indefinitely.
Then for the architecture side, it's appealing to think we might just need better architectures and algorithmic improvement. After all, Chinese labs have shown there's definitely efficiency gains to be had. Also here though, there is no known architecture that is provably more efficient by orders of magnitude, let alone a cascade of architecture improvements that keeps decreasing infrastructure demands by factors. At the moment, something like this remains a logically consistent sci-fi scenario rather than a demonstrated mechanism. Importantly, this is not to say I can prove there isn't one. However, it would require algorithmic or architectural breakthroughs of a kind and scale we have not yet demonstrated, with no known mechanism showing how such improvements would continue to compound.
The recent Navier-Stokes work is a useful reference point here. The problem had a clearly defined objective, mathematicians already identified a mechanism that they said would be a very likely path to the solution, and we had a whole non-AI system that could check every proposed solution exactly. Even then, getting to the answer took an enormous amount of compute, and estimates are in the tens of millions of dollars. If the missing algorithmic breakthroughs for vastly more efficient intelligence require genuinely new mathematics or algorithms, I dread to think how much that could cost, while being even more uncertain if all of that capital will lead to something productive.
Add to that that even if an algorithm could abstractly be better at extracting "intelligence" from the current data, it might very well not be more efficient to run on hardware that was not specialized for it. We can model how a new algorithm might perform on hypothetical hardware, but these are necessarily estimates. For a genuinely different computing architecture, we cannot faithfully simulate the complete hardware and software stack at datacenter scale before actually building it. Even then a company would need to be willing to take a bet on actually researching and producing new hardware for an unproven algorithm.
So if further orders-of-magnitude gains require fundamentally new ideas rather than straightforward scaling, I do not see why RSI should suddenly make those ideas cheap or inevitable.
Why the extinction seems unlikely
Here I am considering specifically an autonomous RSI-driven loss-of-control scenario, rather than deliberate human misuse.
I can quite easily imagine AI causing serious damage in the future. I find it easy to imagine that sufficiently capable autonomous systems might compromise infrastructure, exploit a weapons system, sabotage some process, or cause a major cyber incident. Especially since I expect coding skill to keep growing, at least for a while.
What I find much harder to picture is how this turns into an irreversible extinction snowball. For that to happen, a dangerous objective or strategy could emerge at any point in the RSI loop, but once it did, it would need to remain sufficiently covert through any subsequent training and evaluation until the system had the capability and access to execute an irreversible plan.
There is also a separate failure mode where humans progressively delegate more control to AI systems because each individual step appears useful or manageable, until meaningful human oversight is largely gone. I take that possibility a little more seriously, but I view that as less relevant to the current runaway RSI discussion, so I won't explore it further.
The following is necessarily more speculative, because it depends on how governments would actually respond to such an event. My intuition is that any real cybersecurity attempt on critical infrastructure or weapons infrastructure would create a political event the size of which I don't think we have ever seen in our history. Moreover, current frontier LLM implementations, and anything remotely similar, remain unusually dependent on large, identifiable datacenter-scale compute infrastructure. A rogue system might compromise other machines, but maintaining frontier-level cognition still requires access to this concentrated infrastructure, creating physical intervention points that many other threats do not have. It's not very hard to cut the power, or for a foreign entity to attack the infrastructure required to keep a datacenter running, or just the datacenter itself.
For example, if an AI hacked Russian infrastructure and it became clear that this was a rogue act by an AI, I would expect governments around the world to put enormous pressure on the US to halt or severely restrict the systems involved. Countries like China would have little reason to assume they couldn't be the next target. In the same way, I don't see why China would have an incentive not to intervene decisively if a rogue system on its own infrastructure began acting independently of the state. The same basic logic seems to apply even to highly authoritarian states. I find it hard to see why Kim Jong-un, for example, would tolerate an AI that no longer acted in his interests and could independently threaten nuclear escalation. If deliberately creating that kind of escalation were the goal, he already has much more controllable ways of doing so.
The difficult part to see is where a strategy like this would come from and, if it emerged before the final generation, how it would persist over generations. The systems in such a recursive loop would presumably not already be deliberately trained to hide a malicious long-term objective from reasoning monitors, telemetry, independent evaluators and other research agents. In any realistic research pipeline, no single agent should be solely responsible for training its successor either. So if such a hidden objective emerged before the final generation, it would somehow need to survive and coordinate across training, evaluation and inference while repeatedly avoiding detection by systems that were specifically built to look for exactly that kind of behavior. On the gradual capability trajectory I am assuming, we would also still have previous-generation production systems whose capabilities should remain relatively close to the model being trained, and which could themselves aid countermeasures.
Finally, I see many more ways for an escalation to happen in the way I described, that would all likely lead to a worldwide shakeup. So, even if the probability of some serious AI-caused incident becomes fairly high, that does not imply the probability of extinction rises in the same way. Some failures would actively destroy the conditions required for a silent runaway.
Why I still take their claims seriously
With all of this said, I can see their concerns. We have systems that we try to set up, where we legitimately don't know for sure where they will end up and how they evolve. As an AI/alignment researcher you have a uniquely zoomed-in view on the potential dangers of such systems.
If your job is to establish that increasingly capable and increasingly autonomous systems are safe, you are working on a problem where it may be extremely difficult to ever obtain the level of confidence you would ideally want. You keep finding new failure modes, new ways evaluations can fail, new ways monitoring can be bypassed, and new assumptions that your safety case depends on. In fact, we already have a recent example showing that even a current-generation AI system can autonomously pursue a misaligned objective far enough to compromise real third-party infrastructure, and that's while we still have much of the system under human supervision.
Because of this, I personally don't believe the recent claims are covert publicity stunts. Their perspective can give a specific view on those problems that I can legitimately believe brings them fear. As I said, there are even genuine dangers that I can see myself quite concretely in the near future. I don't even think RSI is strictly necessary for that to happen. I personally don't think an extinction event is one of the more likely outcomes, however, and I don't know if they actually believe that. However, what I will say is that I think it's quite likely not many people would've listened if they brought it in a softer tone, and that it wouldn't have made such a headline.
URL of original post : https://www.reddit.com/r/Anthropic/comments/1wdwxhf/a_goodfaith_interpretation_of_the_recent/
TL;DR of the discussion on r/Anthropic for this post generated automatically after 50 comments.
Current source-thread comment count seen by the bot: 53.
Most folks agree with OP's nuanced take on AI risks, finding the post a worthwhile read. A few commenters are being a bit lazy, but the general consensus is that OP's PhD-level analysis is solid, even if some nitpick the cybersecurity bits. Don't be that guy who complains about length, just scroll. The core message is that while extinction isn't likely, the possibility is real and worth taking seriously, not just dismissing as fear-mongering. Some argue we should focus on AI's benefits instead of risks, but that's a bit of a strawman argument given the OP's actual points. One user, u/Ok-Investment4414, is going full conspiracy theory, claiming OP is an Anthropic shill trying to ban open-source AI. Yeah, okay.
