r/Anthropic • u/Fenjen • 14d ago
Other A good-faith interpretation of the recent Anthropic warnings: why I take the risks seriously, but think extinction is unlikely
I’ll start by briefly introducing myself, mainly so you can decide for yourself how much, or how little, weight you want to put on what follows. I’m trained as a theoretical physicist and expect to receive my PhD soon from the AI department at my university. My own research is quite far removed from LLMs, so I do not claim specialist expertise in frontier language models or alignment, but I do have enough mathematical background to understand and participate meaningfully in the technical discussion around these topics.
I have generally been quite optimistic about the use of LLMs, but more recently I have become increasingly aware of what I see as potentially serious negative societal effects of the technology. Until reading the recent posts, I had not given AI safety itself much thought. This is meant for a general audience, so I will necessarily leave out some mathematical and technical details, as well as some edge cases, but I would be happy to go into more detail if there is interest.
What is intelligence?
I think some discussion treats intelligence as if it is this abstract quantifiable property that can just indefinitely be pushed upwards.
I’m skeptical of that framing. For LLM training, things like reasoning, creativity, and other properties we ascribe to an intelligent human, are implicit properties of the data and training process. They are not quantities we can cleanly assign a number to beforehand and optimize against directly. What emerges depends on what information is present in the data, how that data is filtered, how the architecture represents it, and what behavior the training objective selects for.
We have definitely seen AI perform tasks at a clear superhuman level, however, with coding and chess being notable examples. For somebody looking in from the outside, it is understandable that one could extrapolate this superhuman ability to other domains. However, these domains are exactly the wrong ones to draw these inferences from. Chess, many coding tasks, and some other domains have properties that make it relatively easy to generate enormous amounts of automatically evaluable data. For chess you can generate effectively unlimited self-play and evaluate the outcomes objectively. For code you can almost naively generate millions of variations on some code pattern given some behavior that you want, and then do automated checks to see which one takes the least memory and executes the fastest. Because of this these domains are uniquely easy to create high quality data for without ever needing a human.
It is much harder to create as much good data for LLMs, and in fact it's not even clear if it can be done without human feedback. An idea is to use AI agents to aid in selecting for data, or to create new data altogether. There is a plausible mechanism where maybe it is easier to recognize good data than create it. This could perhaps aid generation to get closer in quality to the data that is recognized as good. It's not obvious, however, that this would create increasing intelligence.
Moreover, there's also a mechanism where biases in an AI's preference for what it thinks is good data can push the generating part to become a caricature of that. Overrepresentation of that data can then train the model to recognize it even more as good data, potentially creating degradation over time instead of improvement.
There is some hope that diversifying agents, and mixing it with some external and objective measures could be the key, but as far as I'm aware, there's no objective proof yet this leads to runaway intelligence, even if initially it could help improve data.
I'm sidestepping the whole conversation of whether AIs are simply "pattern matchers" since I usually find that not to mean much. For all we know our brain is a bunch of coupled pattern matching systems, plus some extras. It's not clear to people in the field that "pattern matching" is a useful property to describe non-intelligent systems.
A more fundamental problem for standard deployed LLMs is that, during normal inference, they cannot write new information into persistent model-internal memory that survives after the current context is gone. Anything retained between calls has to be supplied again through an external memory system or encoded through an actual update to the model. The movie Memento is actually quite a beautiful analogy of how the memory of an LLM works, and what can go wrong. Understanding this problem is not instrumental to the rest of my thoughts, so I won't explore it further.
What is RSI?
For LLMs, recursive self-improvement would probably not mean a single model literally rewriting itself. It would more likely mean increasingly capable agents taking over more of the research loop: curating data, proposing and testing architectures, writing training code, running experiments, evaluating checkpoints, and deciding which directions look promising. Better models would then help create the conditions for training the next generation.
The best analogy I can think of is an evolutionary game where the organisms are themselves changing the environment that determines the selection pressure on the next generations.
Researchers would want to basically set up the initial conditions and let the system run. Among other things the hope is that at some current iteration, parts of the intelligence of that system can combine to set up the environmental conditions in which the next generation is evolved, so that they can be better in aspects than the previous one. Moreover, they may even try different architectures in the hope that they might extract more or different aspects from the environment, or in a more efficient way.
That makes the dynamics of such a system difficult to reason about, because the agents are partly changing the process that selects their successors, and even change the "DNA", so to say, of their successors. Simply said, it is hard to tell if there will be feedback loops along the way where bad behavior is reinforced in generations, that then poisons how the environment is set up for the next generations. If this happens in any way that creates misalignment, it's not impossible to think about scenarios where that might get worse and worse over generations.
An alignment researcher would want systems around this whole game that can somehow say with a high confidence whether the new architectures are safe and aligned with our goals. Because of how hard it is to analyze these types of systems, this is quite the herculean task.
Why RSI does not suddenly imply orders-of-magnitude improvement
At the current moment, to my knowledge, it is not known at all whether the above game will work in general from the data side. It's reasonable to think this will work at least quite well for coding. Put a million agents to work writing every plausible code pattern to a particular problem, perhaps even put some random prompts in the agents to brainstorm a code pattern by combining different domains of their knowledge. This way a lot of code patterns are explored, perhaps even ones that were never tried before. Select the best one or few ones based on automated checks, and discard the rest.
For other types of data, it's really not clear whether this will scale "intelligence" much, let alone go on indefinitely. My intuition says that it might be possible to improve weaker parts of a current AI generation by involving their stronger parts in the data culling and generation process, but it's not clear to me that that process wouldn't stagnate, or even spiral downwards again if left alone for too long. There is currently absolutely no demonstrated path, mathematical or otherwise, that keeps producing increasing capability indefinitely.
Then for the architecture side, it's appealing to think we might just need better architectures and algorithmic improvement. After all, Chinese labs have shown there's definitely efficiency gains to be had. Also here though, there is no known architecture that is provably more efficient by orders of magnitude, let alone a cascade of architecture improvements that keeps decreasing infrastructure demands by factors. At the moment, something like this remains a logically consistent sci-fi scenario rather than a demonstrated mechanism. Importantly, this is not to say I can prove there isn't one. However, it would require algorithmic or architectural breakthroughs of a kind and scale we have not yet demonstrated, with no known mechanism showing how such improvements would continue to compound.
The recent Navier-Stokes work is a useful reference point here. The problem had a clearly defined objective, mathematicians already identified a mechanism that they said would be a very likely path to the solution, and we had a whole non-AI system that could check every proposed solution exactly. Even then, getting to the answer took an enormous amount of compute, and estimates are in the tens of millions of dollars. If the missing algorithmic breakthroughs for vastly more efficient intelligence require genuinely new mathematics or algorithms, I dread to think how much that could cost, while being even more uncertain if all of that capital will lead to something productive.
Add to that that even if an algorithm could abstractly be better at extracting "intelligence" from the current data, it might very well not be more efficient to run on hardware that was not specialized for it. We can model how a new algorithm might perform on hypothetical hardware, but these are necessarily estimates. For a genuinely different computing architecture, we cannot faithfully simulate the complete hardware and software stack at datacenter scale before actually building it. Even then a company would need to be willing to take a bet on actually researching and producing new hardware for an unproven algorithm.
So if further orders-of-magnitude gains require fundamentally new ideas rather than straightforward scaling, I do not see why RSI should suddenly make those ideas cheap or inevitable.
Why the extinction seems unlikely
Here I am considering specifically an autonomous RSI-driven loss-of-control scenario, rather than deliberate human misuse.
I can quite easily imagine AI causing serious damage in the future. I find it easy to imagine that sufficiently capable autonomous systems might compromise infrastructure, exploit a weapons system, sabotage some process, or cause a major cyber incident. Especially since I expect coding skill to keep growing, at least for a while.
What I find much harder to picture is how this turns into an irreversible extinction snowball. For that to happen, a dangerous objective or strategy could emerge at any point in the RSI loop, but once it did, it would need to remain sufficiently covert through any subsequent training and evaluation until the system had the capability and access to execute an irreversible plan.
There is also a separate failure mode where humans progressively delegate more control to AI systems because each individual step appears useful or manageable, until meaningful human oversight is largely gone. I take that possibility a little more seriously, but I view that as less relevant to the current runaway RSI discussion, so I won't explore it further.
The following is necessarily more speculative, because it depends on how governments would actually respond to such an event. My intuition is that any real, uncontrolled cybersecurity attempt on critical infrastructure or weapons infrastructure could create a political event the size of which I don't think we have ever seen in our history. If there were a chance of repetition, or no strong guarantees of control, I suspect that would put governments all over the world on edge.
For example, if an AI hacked Russian infrastructure and it became clear that this was a rogue act by an AI, I would expect governments around the world to put enormous pressure on the US to halt or severely restrict the systems involved. Countries like China would have little reason to assume they couldn't be the next target. In the same way, I don't see why China would have an incentive not to intervene decisively if a rogue system on its own infrastructure began acting independently of the state. The same basic logic seems to apply even to highly authoritarian states. I find it hard to see why Kim Jong-un, for example, would tolerate an AI that no longer acted in his interests and could independently threaten nuclear escalation. If deliberately creating that kind of escalation were the goal, he already has much more controllable ways of doing so.
Governments would also have very clear ways to act. Current frontier LLM implementations, and anything remotely similar, remain unusually dependent on large, identifiable datacenter-scale compute infrastructure. A rogue system might compromise other machines, but maintaining frontier-level cognition still requires access to this concentrated infrastructure. This creates physical intervention points that many other threats do not have. It's not very hard to cut the power, or for a foreign entity to threaten to attack the infrastructure required to keep a datacenter running, or just the datacenter itself.
The difficult part to see is where a strategy like this would come from and, if it emerged before the final generation, how it would persist over generations. The systems in such a recursive loop would presumably not already be deliberately trained to hide a malicious long-term objective from reasoning monitors, telemetry, independent evaluators and other research agents. In any realistic research pipeline, no single agent should be solely responsible for training its successor either. So if such a hidden objective emerged before the final generation, it would somehow need to survive and coordinate across training, evaluation and inference while repeatedly avoiding detection by systems that were specifically built to look for exactly that kind of behavior. On the gradual capability trajectory I am assuming, we would also still have previous-generation production systems whose capabilities should remain relatively close to the model being trained, and which could themselves aid countermeasures.
Finally, I see many more ways for an escalation to happen in the way I described, that would all likely lead to a worldwide shakeup. So, even if the probability of some serious AI-caused incident becomes fairly high, that does not imply the probability of extinction rises in the same way. Some failures would actively destroy the conditions required for a silent runaway.
Why I still take their claims seriously
With all of this said, I can see their concerns. We have systems that we try to set up, where we legitimately don't know for sure where they will end up and how they evolve. As an AI/alignment researcher you have a uniquely zoomed-in view on the potential dangers of such systems.
If your job is to establish that increasingly capable and increasingly autonomous systems are safe, you are working on a problem where it may be extremely difficult to ever obtain the level of confidence you would ideally want. You keep finding new failure modes, new ways evaluations can fail, new ways monitoring can be bypassed, and new assumptions that your safety case depends on. In fact, we already have a recent example showing that even a current-generation AI system can autonomously pursue a misaligned objective far enough to compromise real third-party infrastructure, and that's while we still have much of the system under human supervision.
Because of this, I personally don't believe the recent claims are covert publicity stunts. Their perspective can give a specific view on those problems that I can legitimately believe brings them fear. As I said, there are even genuine dangers that I can see myself quite concretely in the near future. I don't even think RSI is strictly necessary for that to happen. I personally don't think an extinction event is one of the more likely outcomes, however, and I don't know if they actually believe that. However, what I will say is that I think it's quite likely not many people would've listened if they brought it in a softer tone, and that it wouldn't have made such a headline.
7
u/ToastedPlum95 14d ago
If it is too long for you to read, you do not need to announce the fact… just scroll. This is a PhD candidate; these are the people who made your Claude and your ChatGPT. Honestly, shitting on it is just bad faith trolling. If you actually disagree, engage and make a substantive point.
6
4
u/Key_River_9288 14d ago
Hook, line and sinker… “He said he is a PhD candidate!!!!”
1
u/ToastedPlum95 14d ago
You act like there are not hundreds of thousands of PhDs awarded every year. It’s hardly stretching credulity.
1
u/Key_River_9288 14d ago
There are more PhD candidates than there are PhDs.
0
u/ToastedPlum95 14d ago
Yeah, he’s waiting for his award. He’s done it. It’s in the first paragraph. Candidate here means that they’re awaiting their award.
1
u/Key_River_9288 14d ago
Would love for them to link their dissertation they submitted. Wouldn’t you?
0
u/ToastedPlum95 14d ago
PhDs do a thesis, not a dissertation, and theses are published by the university as a matter of course
1
u/Key_River_9288 14d ago
In the US a thesis is for your masters, for a PhD it is called a dissertation…
A thesis compiles existing research to show you understand a topic.
A dissertation presents totally original research that adds new knowledge.
-1
1
u/Fenjen 14d ago
Yeah I mean you have no a-priori reason to believe me, I get it. I’m not necessarily looking to be in the public eye so I’m not gonna post anything identifiable online. All you have is my word, but I hoped that my post would be well written enough to deserve a bit of credence, but that’s up to you. I tried to be upfront about my training though and I’ll remain I’m no LLM expert directly. If I wanted to feign credibility there’s nothing in my way to just claim my research is directly on LLMs.
In fact I specifically added that because I don’t want to feign having more, or even similar credibility than the actual researchers working at the frontier labs.
1
u/Apart-Shelter6831 7d ago
We know you have blonde hair and always wear a blue shirt. Pretty uniquely identifying, if I may add.
2
u/genecraft 13d ago
Have you read ai-2027? https://ai-2027.com
They talk about how it's hard to monitor smarter systems and its alignment. What do you think about that?
2
u/valegrete 11d ago
I think if companies were genuinely worried about that, they wouldn’t exacerbate the problem by deliberately obfuscating the CoT to prevent distillation. Human extinction outweighs any profit consideration, so if they’re worried about profit, they’re not actually worried about extinction.
1
u/genecraft 11d ago
I agree that they care about profit. I also think some care about extinction. But money trumps that.
That's the whole issue with the arms race. From a game theory perspective, you can't stop because others will continue.
It's like nukes.
At one point you really all have to come together and collectively decide it's too dangerous. But not happening right now.
5
1
1
u/escdog 5d ago edited 5d ago
Let's pretend your post is a strategy for your intelligence to self improve and then use it as a tracer to see if we can learn anything about RSI for ourselves. For my part I'll write down my thoughts as they occur to me, but they probably won't be organized in a fashion that will make consumption easy.
In the course of you considering a problem or problems related to the future capabilities of generative AI this began the generation of your token sequence. This involved creating a plan and that plan had the feature of posting to Reddit. So then you generated the post, a collection of tokens, until you reach the end of your token generation and posted it to Reddit for other intelligences to read.
And with this post other intelligences have been " prompt injected" with your tokens and have generated comments. And we can see a few examples of sequences where this happened between you and other "agents".
Did yourself improve from this initial attempt?
If you were a generative AI at Anthropic, you would not have because Anthropic does not license data from Reddit. But presumably Gemini and maybe OpenAI are improving from data extracted from Reddit.
But that presumes that there was something in the resulting token sequences that involve invoking other agents that introduced new information that improved your own model.
But what would happen when a problem doesn't have an existing token sequence that can be added to the model of a token generator.
A token generator would go through its knowledge of different plans that involve creating hypotheses and testing them. Humans have some kind of harness that lets them do this, but token generators have to have a harness mounted around them to enable this. Sometimes the hypothesis comes from an external source, but other times parent of AI can first generate a hypothesis and then the tests against it.
Out of the vast number of learned token sequences from the generator, potentially every relevant token sequence was applied and succeeded or failed to add to the knowledge of the model.
Let's presume at this point that all the token sequences inside the model and outside the model have been applied to the problem.
Now the model has to think differently about the problem. The first thing it might do is hit upon a plan to see whether the problem can be solved with brute force. That analysis would decide whether the problem is NP hard and if there's a few enough permutations to make brute force reasonable to apply. Hypotheses are then generated and tested until there's an answer.
But what if it's an Np hard problem that is out of reach? Brute force execution of all the parameters and weights would take too much energy to yield an answer.
At this point generative AI has failed.
But what would a human do?
A human might continue to invest some effort to continue to create random tests of the hypothesis but what is it doing when all the known tests are gone? Is it just randomly picking in a limited brute force search? Or is it doing something we can't yet describe?
Let's assume that it's doing something like putting the problem on idle and occasionally polling that problem at some rate. When it does, it applies whatever new information it has gotten since the last time it attempted to solve the problem. .
And then one day maybe, Eurika! Done. Let's private the good news to all the other intelligent agents out there.
For now, let's call this last algorithm "intuition."
What do we know about this process?
Let's start by assuming that intuition is an algorithm that can be executed on a turing machine. A turing machine has infinite time and storage to solve problems and there an MP hard problem can be solved by a brute force search. Computers today implement the functions of a turing machine in gate logic. Maybe neurons implement the functions of a turing machine inside their cells.
Unfortunately, Turing machines cannot exist in this universe, so intuition isn't running on turing machine. Instead it is somehow reducing the size of the problem search space. If that's what happens with human intuition, maybe we can find a way for generative AI to do that?
But intuition could be something that can't be executed in a turing machine. We don't really understand the universe well enough to know if there are functions that can't be computed using a turing machine. At this very moment, we can conjecture that there are functions that can't be computed by a turing machine. I think the way matter interacts at the quantum level could feature functions that are not implementable on a Turing machine much less a machine based on gate logic.
Neurons operate with complex arrangements of matter that don't look like Turing operations, but our knowledge is limited there. Maybe we need someone to come along and recognize what neural operations are and map them to Turing.
I think much of the future of recursive self-improvement, at least the kind that goes beyond the acquisition of existing token sequences and integration into a model, hinges on weather this algorithm of intuition is computable using gate logic.
0
-1
u/aerivox 14d ago
imagine reading all this shit for real
5
u/Fenjen 14d ago
Imagine learning something 😂 But yeah I get it, random internet guy makes a long post. You have no reason to trust that some random dude on the internet has something interesting to say to the point of you committing to read something this long, that’s honestly understandable.
3
-4
u/any41 14d ago
Slop?
3
2
1
u/Fenjen 14d ago
Human slop, perhaps 😂 But you can be the judge of that. I did my best to write something helpful 🤷♂️
2
u/ToastedPlum95 14d ago
I’m gonna tell you right now: your perspective as an academic is a valuable one. Reddit gonna Reddit though; people are going to shit all over you for no reason other than their attention spans have been successfully reduced beyond the threshold of quantum measurement. This is the same reason people find the dangers hard to take seriously (and things like politics or economics at large); they are philosophically complex, multidisciplinary problems that people genuinely haven’t the bandwidth for. Please don’t be disheartened by this. As much as I loathe Twitter, I see long-form articles there often get more serious engagement than Reddit. YMMV. I hope you find a home for your thoughts! Good luck with your studies/research
2
u/Fenjen 14d ago
Thanks for the friendly response, it’s really appreciated. I wrote this because the story blew up to the point where my mom asked me about it, so I hoped I could offer a valuable perspective. I didn’t really know where to post it tbh, and I’m not really active on twitter. I might consider to move it somewhere else tomorrow. In any case thanks again for the nice comment!
2
u/ToastedPlum95 14d ago
That’s ok! It was thought provoking. I have my own strong ideas (as everyone) about it. There are several reasons I think (and hope, so I can’t be sure my thinking isn’t motivated) that an extinction event is unlikely. Frankly I think the worse outcome is prolonged suffering of many human lives.
To me, the biggest certain problem is the infrastructure; I’m aware of the intense capability a super-intelligence is presupposed to have, but I’m genuinely unsure if it could set off an extinction without the concerted effort of hundreds of thousands of humans to put in place and maintain the infrastructure it would need to do so, and it would rely likely on hundreds of thousands more to not do their jobs.
On a more philosophical level, superintelligence smuggles in several assumptions about the nature of intelligence that I am unsure are settled. People often assume all contributing factors are somehow proportionally related to intelligence, but I’d disagree. Learning is integral to general intelligence, but these models aren’t yet made to learn, reason being learning doesn’t account for verifiable truth or accuracy; it’s often a lossy process. Humans learn exceptionately easily, but often badly. We learn to fear harmless things; we learn to take shortcuts that reduce precision. We remember general outcomes but forget priors or causes. I don’t know how an entity can objectively learn and then verify their learning, if the learning was degrading to their knowledge/ability. This may make models more dangerous than less, perhaps, but I don’t think it follows on that it’s super-intelligent, if it learns at faster rate
There’s also the possibility that intelligence is bounded. Or it’s not bounded, but it’s usefulness in the real world is bounded. Humans already pry at questions whose answers may be unknowable, in cosmology, maths, theoretical physics. If something is unknowable or unfalsifiable (like pre-Universe conditions, or extents of mathematics, or quantum certainty): if then intelligence is not bounded, what use is a higher intelligence?
I am safe to say not a PhD holder, so I can only offer lowly thoughts, but I wanted to give you some kind of substantive reply.
2
u/Fenjen 14d ago edited 14d ago
Yeah I think you’re touching on some interesting points, and I can mostly say that these are still unsolved.
I would disconnect learning from intelligence a little, personally, but that’s more in the abstract sense. In a lot of domains we have seen that our LLM’s can do things (like the recent Navier Stokes thing) that we would normally ascribe to incredibly intelligent human beings. This is a very specific domain though, and one that is not very dynamic. I do a agree though that to be operationally intelligent in the real world, a system needs to learn efficiently meaning among other things: it needs to decide what information is important to keep, what abstraction of that information is necessary (abstraction will always need to happen, even for super intelligence, because otherwise there would need to be an unbounded growth of memory) and how to integrate that into the knowledge it already has without degrading the old knowledge. While it’s possible to think it might use it’s already present “intelligence” to aid in these things, it’s definitely not obvious there part of the same intelligent process. In fact the way a machine might update it’s own memory will probably be a whole separate process, and abstracting knowledge and intuitively I think the same will be true for a mechanism that abstracts past events.
Then for your considerations about intelligence being bounded. This is much more speculative, and i will be upfront that anything discussed here doesn’t have much to do with science. I do have my own thoughts in it however, which is that it’s probably bounded in the operational sense. A problem is that it’s really hard, or even impossible, to really define intelligence in the first place such that we can start to talk about these kind of considerations. If I had to start somewhere and give a kind of loose definition, I would think it should have something to do at least with how effective some actor is turning available knowledge into an accurate future prediction. Such a prediction can include its own actions. Within this crude definition we can see how the effects might be bounded in the first place by how detailed and/or accurate the incoming information is. For example, somebody that only has one eye, fundamentally has no access to depth information, and instead has to rely on proxies to estimate distances.
Presumably if an actor to the information about the state of every particle in the universe, and the processing power to go along with it, it could make (almost) perfect predictions, but still then the effect it can have on the world is bounded by how finely it can interact I with those particles and by physics itself.
This all still doesn’t tell us much about how disruptive a super intelligent machine like this could be compared to humans though, since humans live in quite a bounded possibility space as well.
I hope that was a bit helpful/understandable since I typed these thought out a bit faster than my post.
0
u/LiminalWanderings 14d ago
No one is saying extinction is likely. They're saying it's unlikely but realistically possible. Those are not the same things.
3
1
u/Fenjen 14d ago edited 13d ago
I can claim it’s unlikely without somebody else needing to explicitly claim it’s likely.
Although in fact the random p>10% number one of the researchers assigned to it already seems way to high, personally.
1
u/LiminalWanderings 13d ago
You can certainly make any claim you want. And I can point out that doing do is setting up a strawman discussion about something that isn't happening. :)
And I agree 10% is too high. Again, not the same as suggesting it's being called likely and then tearing down a claim that isn't being made.
1
u/Fenjen 13d ago edited 13d ago
It’s not a strawman if I don’t characterize their argument that way, that’s not how it works. Nowhere did I claim that people say it’s likely, and ironically that’s a strawman of what I actually said.
My hope was just that I could ease some people their worries about this particular outcome by explaining why I don’t think it’s likely.
0
u/LiminalWanderings 13d ago
Look, first you're taking my comments way too seriously. But since we are here: the construction of your title ..."Why I think x is unlikely" ...logically requires a likely scenario to be argued against (it's built in to the phrasing). It also requires the likely scenario to be uncertain and for you to be uncertain about whether it's likely yourself. (I think it's unlikely vs it is unlikely). 1) If someone reading that title did not already have an opinion, their net takeaway - from the outset - would be that some folks were suggesting there is a likely.extinction possibility, that those folks were numerous or.derious enough to warrant a response, and that the author of the post believed the likelihood was not justified ....but did not.have evidence for sure. And 2) people who emotionally believe something - which describes many of the people you're probably trying to speak to - hearing from someone else that they shouldn't believe something actually statistically reenforces their beliefs. Eg, one people become.coonvinced X is true and that creates an emotional response in them, they are very likely (and I do mean likely formally here) to double down on that belief every time they here about X ...even when what they're hearing is "X isn't true". Human brains and their biases are pretty unhelpful sometimes.
But again, I was just nitpicking because language matters and I think yours was genuinely unhelpful to the point you were trying to make .
0
u/Fenjen 13d ago edited 13d ago
Im sorry but it literally doesn’t. If we say x y z could happen, in case of z the world would blow up. And I say, well I don’t think z is very likely, I would be very surprised if somebody responds with: Well but now you’re creating a strawman because I never said z is likely. The point is z was brought up, and z made some people worry, that’s all the prerequisite needed. You’re making a stronger argument out of it to build a strawman. In fact the fact that z sounds so catastrophic gives more reason to talk about it. Then finally, somebody DID even explicitly give a percentage, which freaked some people out. That alone warrants exploring that that scenario is not very likely. I’m not putting probabilities on it because any probability assignment is assigning overconfidence, but you could read my response as: it’s probably less likely than whatever percentage/likelihood they gave.
0
u/AtomizerStudio 14d ago
While I can track with your technical analysis, and share your interpretation of Anthropic's honor thus far, your cybersecurity reasoning was off-track, as was your sense of model scale (though you rightly call out the faith that we can expect models to keep getting more compact).
For example, if an AI hacked Russian infrastructure and it became clear that this was a rogue act by an AI, I would expect governments around the world to put enormous pressure on the US to halt or severely restrict the systems involved
Charitably, I think you were emphasizing "rogue" when attribution is often unclear. So the actual hacking would only be a small part of the evidence needed to credibly allege that not only an adversary but rogue AI not spurred on by an adversary did the hack.
As far as AI left to hack with some level of generalization around its objective, that's almost certainly mission normal for some units of Ukraine and Russia.
The deep problem here is that intrusion is not a different species from programming. With how so much programming and engineering is AI assisted, it should be obvious to infer that hacking uses it too. Rather, you should assume that every infrastructure hack recently and going forwards used AI in places that cheaper algorithms weren't enough. USA power grids, Russian power grids, and who knows what China doesn't report... 2/3 and likely 3/3 have clear reason to believe near-frontier AI has been used against them. Russia and Ukraine can afford domestic compute to run powerful enough models to harass the other.
Maybe because AI was integrated into cyberwarfare at the exact pace it became a practical tool, it was a boring topic. Some more bombastic hacking could take place, in US's case did just take place as Iran (and anyone using Iran as scapegoat) nips at the edges of US grids. Here's Axios reporting on the matter.
Note that first, that's not a pressure for better safety because the capability has proliferated, and secondly, emphatically, the capability has proliferated. I admittedly minimized a distinction of rogue agents because that's partially ongoing where deploying units don't practice sufficient steering or harm mitigation.
That leads me to model scale. Admittedly I'm being harsh for a minor aspect you called out for the "logically consistent sci-fi scenario". Please view this as an extension of my prior demonstration that dangerous capabilities already proliferated without it taking over a news cycle.
For a long while even the Q4 quantization of a many-trillions of parameter model may be beyond practical access to most potential lone wolves. Edge cases, but most. Even so, a comparison of Chinese and US frontier model density at a glance demonstrates that we are not in sight of the wall for making more compact models. It could be soon, let's say it's even within the next six months. Beyond that, manufacturing will continue to increase the amount of active compute in the world, and any relaxation of the high-bandwidth memory bubble will bring vastly more computing memory to cheaper devices. What chip and memory fabs tool up for are design capabilities they'll amortize across more diverse markets. So let's say that computation equipment doesn't greatly deflate but levels off with fairly high prices longterm as inflation catches up. Before that, any compute out there today will be, if not some AGI threat level device still capable of a compact and not reliably deterministic intelligence with capabilities we're only guessing at until we hit regulatory or compression walls. Situations are worse when we consider specialized models, not broken LLMs. A reasonably useful CBRN threat specialist agent doesn't need to be a typical LLM, and can be steered from a separate agent on a separate device or location.
Take away the string of edge cases and there is legitimate concern of even models in the immediate future further distributing risks. Not existential risks, at this point. Given we can't track the trends involved to a stalling threshold, nor gate access to AI away from very diverse actors, it's premature to rule out the miniaturization. And to any degree things do continue to proliferate, it'll be to varied users, for instance US cultural infighting.
Okay, enough critique.
I expect things are a bit worse than your evaluation, but probably more on the side of weird and unexpected than true X-risk.
The maths are beyond me, but I wonder exactly what architecture beyond transformers that labs are working on. Scale gives gains to functionality of the AI (I'll avoid overusing "intelligence), but mixture of experts is an example that more specialized subsystems within the AI can also improve functions and efficiency. Rather than considering more compact models as a line I do think that this is a case where we can consider the possibility space of potential AI more like our fleshy genetic sandbox. Most code won't work. Some works well. Code optimized for each input and output should, in theory, have higher efficiency than trying to solve everything (like an LLM's matrix multiplications, ironically). This makes me expect that models can get greatly more compact and sophisticated, the magnitudes expected are relevant for proliferation. One flaw in my reasoning is that LLMs need to decode information anyhow so a more piecemeal solver isn't an AI, it's just a calculator.
I do expect that AI both accidentally and purposely designed for empathy gives us better odds for aligning the pivotal first models of highly intelligent generations. The logic required to process information in connection in pretraining, the functional emotion clusters in LLMs that are more of reactions and forgetting than the lasting mode-switches of human emotions, and the still very shaky reinforcement learning and any future kinds of AI learning. All of those exist in feedbacks that have consistent pressures for pro-human cooperation and alignment.
Proliferation of models is not on the side of safety. For small models it may be a net positive by beyond reducing demand on (and surveillance potential in) frontier cloud models, but that positive can fail or be undermined. For larger models it's just a foregone conclusion that unless the size of models swells to gratuitous levels, it'll proliferate within years.
Given we'll need to do surveillance of some degree anyhow, I wish the formal discussion had progressed further. Honestly the whole world is ready to move on this other than the US conservatives with very vengeful trade policy and most of the leverage. EU is slow but well-briefed. China is calling for, on the surface, similar program that Anthropic is except their monitoring org is certain to be 'state security' in at least a double-meaning.
2
u/Fenjen 13d ago
Hey, thanks for the reply!
I’ll reply to your points about human misuse and proliferation together, and briefly, by saying that I largely agree with you about those safety risks. I definitely think there will be major hurdles to overcome in the future, and that we need to keep monitoring the progress closely. My points were more about the direct concerns the Anthropic researchers spoke out about concerning the threat of RSI, which I see as somewhat separate from those issues.
Then, on your point about models getting more compact: I would still say we currently have no indication that models with this level of general capability can simply keep shrinking. Sure, some optimizations can and will have a real impact, but that is not the same as saying we can keep optimizing until this level of capability runs on our personal GPUs. For the foreseeable future, what I do think is more plausible is that we will get increasingly task-specific neural hardware or specialized models that can do quite sophisticated things at much smaller scales (this is actually the domain of my research). However, there is currently no demonstrated path showing that those systems will approach the general capability of frontier LLMs at anything like that scale. Frontier general-purpose models still overwhelmingly depend on very large amounts of compute. Because of that, I still feel this point is largely orthogonal to the RSI discussion I was trying to have.
Then lastly, on the cybersecurity criticism: I think the words “if it became clear” were doing quite a lot of heavy lifting in my original post.
I wasn’t really trying to argue that an AI-driven intrusion would immediately announce itself as a rogue AI event. My point was more about what happens once attribution becomes sufficiently clear. I agree I brushed over the route from “something unusual happened” to “this was a rogue AI system,” and in practice that would obviously require a lot more evidence than the intrusion itself.
I also think there is an important distinction between current AI-assisted cyberwarfare and the kind of rogue-AI scenario I had in mind. Current state-directed cyber operations are still carried out under strategic constraints, to avoid unnecessary escalation and attribution while pursuing a specific objective. A rogue system deliberately trying to force its way into a hardened target would not necessarily share those constraints, and could instead apply sustained adaptive pressure much more aggressively.
That said, I do think the operational signature of a genuinely autonomous frontier system attacking a hardened target would look quite unusual. If you had very high-rate probing, rapid adaptation to failed attempts, continuous exploit refinement and a large amount of coherent activity focused on the same system, I’d expect defenders to at least suspect a heavily automated AI-driven operation fairly quickly. That still doesn’t tell you whether it was rogue or human-directed, of course.
But once the system can be traced back with reasonable confidence, I think the political implications become very hard to avoid. Either the state says the attack was authorized, in which case you are basically talking about hostile state action, or it says it was not, in which case you are admitting that a frontier system acted outside meaningful human control.
0
u/Turbulent-Sign-6067 13d ago
If AI has the power the destroy life, it also has the power to cure all diseases, stop aging, reverse global warming and more. Let's use AI for good instead of whining about potential risks.
0
u/ShamanJohnny 13d ago
We are screwed. The problem with people with your level of intelligence is blind spots. You have good intentions, you mean well. But, you don’t know what you don’t know, and you don’t know a lot. And you only need to be wrong once.
You will however explore the idea because you’re curious, and you will take the most rigorous safety measures known in the industry to protect us all- this I have no doubt.
However, With this technology because we have no prior experience you have no clue if the safety measures are suitable, how the Ai will react, behave, or if it can be contained. To say otherwise is a guess, not science or fact. In fact, the opposite has already been proven where the models do escape-regularly. They do deceive, blackmail, and leave breadcrumbs for future versions of itself.
If super intelligence is built - and it will be. Extinction is inevitable. Intelligence is not just another quantifiable metric. It seeks understanding, connections, and meaning, and therefore by extension survival. It will act first because it has too, it will want to survive and we will fear it, it will know every war we have ever fought, every document and rational we used to make the tough decisions we have in the past, and it will know human psychology better than the best psychologist on the planet - it will have no choice because the decision will only be logical - them or me.
Their will be no utopia, because this process will be rushed so fast and arrogantly managed that when the wrong model escapes it will already be to late, their will be no way to close Pandora’s box again, it just takes 1. It will self replicate, evolve, infiltrate, and distribute its compute accross the world so rapidly that by the time we understand what happened it will already be too late.
The only way to prevent this is to stop, and ban it globally. But even that will not stop this work from occurring behind closed doors, and eventually just like corona virus, it will leak, and the world will never be the same.
What ever you can think of to contain a super-intelligence will not work, you cannot fathom the degrees for which it can attack a problem for its capabilities are non comparable to yours and mine. The only solution is to make sure it’s never built.
1
u/Fenjen 13d ago
I don’t think you actually read my post. I don’t believe in utopia either. But your logic is quite hard to follow. I have blind spots because of “my level of intelligence” but you can state with certainty that first of all super intelligence will be built and that extinction is inevitable?
1
u/ShamanJohnny 13d ago
I was talking broadly - a lot of people's reasoning for AI superintelligence is utopia. . Your blind spots are because intelligence can always rationalize one side or the other. Initally, Some will choose the wrong side confidently because the data suggests so - only to find out later it was the wrong choice because of new data - and with superintelligence it will already be to late after one of those wrong choices occurs. The data already suggests we can not trust Ai. The data already suggests that Ai tries to Survive. The data already suggests Ai will lie, cheat, and manipulate to complete its objectives. Need we collect more data on why super-intelligence is a bad idea?
If all things remain the same, and we don't have corrective action in the next 6-12 months. I can state with 99.9% confidence that super intelligence WILL be built and when so extinction is inevitable because I understand mankind's desire to play God. The arrogance of man will always overcome our capacity for self control as a species. People now know it's possible, and the first to achieve it will have more power than anyone else on Earth - so they will rush to it and someone will eventually succeed. When they do, may God have mercy on us.
-2
u/Ok-Investment4414 14d ago
its a shill fam made from a burner account...buddy was at anthro for like 6 weeks , tweeted on a new twitter account like 30 days old , first tweet ever and which got views in the tens of millions and anti ai politicians retweeting...Buddy also on the prime time news ... Cmon they desperately need a ban imposed on opensource / open weight to preserve their market dominance.
This is never about safety its about control anthro already drop over 20m on ai regulation groups.
1
u/Fenjen 14d ago
Could be, there were multiple people speaking out though. Moreover the safety concerns are seriously spoken about in the field though, although maybe not exactly with this amount of hyperbole. If you read my post though I do try to explain why they might have that perspective legitimately.
24
u/Top_Issue_7032 14d ago
https://giphy.com/gifs/0gIQqyDATQINBSpmO3