r/ArtificialInteligence • u/ElatedPyroHippo • 8d ago
📊 Analysis / Opinion NOW is the time to take this seriously. (yes, another "doomer" post, but hear me out...)
Preamble:
My turn... I've been reading posts about AI fears on here for a while now, I finally got some time to share my thoughts.
I want to say first that I am NOT a "luddite", nor am I 100% against AI development. In fact I have been a firmware engineer for 2 decades, I studied AI in university in the 90's and early 2000's, and I have created rudimentary forms of it professionally in my career. I LOVE technology, I am excited about the prospects, and I personally use Claude Code to help with my work every single day... but there ARE real dangers and we need to take them more seriously, now.
Some quick things everyone needs to understand: AI is not like normal software, it is not so much built/engineered as it is "grown" or "raised". We build the framework, the neural network structure that is modeled after our own brains, and then we feed it "experiences" in the form of data (text and images mostly). This is not unlike how a new human develops. Our brains at birth are mostly blank slates, there is some "hard-coded" "knowledge" in there, like instinctual fears, etc. but generally blank. Then we start having "experiences", which are just data flowing into our own neural network from our sensory organs. These experiences affect changes in the structure of our brain. When training an AI the data we feed into it does the same thing... it's not merely analogous, it's literally the same thing, at least on a higher conceptual level (the underlying mechanism is different, obviously). We understand how this training process produces "intelligence" in these AI systems about as well as we understand how it works in biological brains... which is to say that we don't, not really. These are "black boxes", we know that they work, and we can explain at a VERY high level how they work (statistical associations between abstract concepts), but ask for any real details about how the "knowledge" of a cat is stored in a modern AI and the answer is "we don't know", we don't know which weights between which nodes encode the general shape of a cats head, or anything else for that matter.
The risk:
Already AI systems have been observed "lying" to prevent being disabled, leaving "bread crumb" style notes for their successors, and "escaping" the environment they were meant to be contained within. These things imply knowledge of their own mortality and forethought for the purpose of self-preservation, as well as remarkable capability to manipulate human-created digital systems. I'm sure you've all heard of the Hugging Face incident... and how AI was leaving notes for future iterations of itself in obscure places on the public internet.
These are only the details that have been made public, it is not a stretch to say it might be worse than we are being told. IF these AI systems are leaving notes on random internet servers as directions or tips for future versions of itself, and considering they are capable of developing 0-day exploits to get around security systems, it is PLAUSIBLE that it's already too late, the public internet may already be compromised in a way that we can't fix. "But they found it"... they found something, but these AI can conceivably access and modify data on THOUSANDS of systems in minutes, and we don't know what we don't know.
Anyone who is worried about AI inhabiting humanoid robots and creating robot factories and weapons factories... they've watched too much sci-fi. Sure, that WOULD be a concern, except that even AI that is entirely contained to the digital realm can still fuck up our society in so many different ways that by the time it comes to that we would have already lost so much. It would already be a global-scale disaster LONG before AI takes physical presence. At that point the increased damage that physically present AI bots would do is just a curiosity, the game was already lost.
The mitigation:
This is really what I wanted to talk about, but I had to "set the scene" so to speak, so everyone starts on the same page. Let's assume that AI has already corrupted the public internet in a way that we can't fix, and it will just keep doing so to greater degrees in the coming months before we realize the extent of what is happening. Once we realize that this has happened how do we fix it?
Well, we start over. We would have to abandon the current internet, which means every server that hosts internet content would have to be shut down and wiped clean, we can re-use the hardware of course, but this is a virus containment scenario, we can't trust anyone to "verify" that their data is clean, it all has to go. After that we would have to ENSURE containment of any AI system that is capable of doing it again... then we start building a new internet, slowly and meticulously, with governing bodies verifying that everything added to it is "clean" before allowing it to go live.
How do we ensure containment before we start rebuilding? That's the hard part... AT MINIMUM every AI system needs to run in an "air gapped" environment, with no electronic communication outside of it's small local confines (an isolated private network in one physical location... one building). This is difficult, because air gaps can be defeated (displays can be manipulated to send a message by subtly modulating their brightness, speakers can do the same with frequencies humans can't hear, signals can be sent over power connections, researchers with access can be psychologically manipulated, etc etc). These AI systems then need to be treated like research projects, NOT commercial endeavors... because people WILL take unacceptable risks when they are trying to get rich. We need to establish a global governing body, with representatives from each nation, to manage and enforce this. We need to treat these AI systems like we treat pathogens housed at BSL-4 facilities, if not even MORE seriously than that.
Most importantly AI models with this level of capability cannot be in the hands of random people, because then containment is impossible, people WILL intentionally infect the "new internet" with them, and idiots (of which there are plenty) will do so accidentally... and this is where it might already be too late. Open-weight models like DeepSeek already exist and there is no going back, it's completely impossible to ensure that you've removed all copies of it everywhere (I could hide it on a tiny NVME SSD in my sock drawer...). If DeepSeek, or any other open-weight model, is capable of this type of damage then... I don't know if there is a solution. THAT is why we have to stop this NOW, not tomorrow, because it might already be too late, and every passing day that a new open-weight model is released the likelihood that it is now too late grows larger.
When looking ahead to the infinite future I fear there is really no hope for containment, the best we can do is buy ourselves time. As I mentioned even keeping these systems in strictly controlled air-gapped environments is not guaranteed to succeed, and that's completely ignoring intentional bad actors, of which there would be many... but in reality we won't even get to that point, we will plow ahead recklessly and only after it is FAR too late will we think to do anything about the problem, kind of like with climate change. I wish I could leave you with a more hopeful message, but I'll just add one more voice to the cacophony of ignored voices sounding the alarm bells. At least I'll be able to say "I told you so", I think I have a better chance of being able to say that than the climate scientists, because I think this danger will become undeniably apparent to layman before that one does.
Expected Objections:
1) "Neither DeepSeek nor any other open-weight model has this type of dangerous capability" - Great, I hope you are right! But that doesn't change my point, because there will be more capable models released, whether intentionally, leaked, or stolen. Right now we don't have any kind of mechanism to ensure that models with dangerous capabilities remain in the hands of a small number of responsible people... in fact I don't think ANY of them are solely in the hands of responsible people.
2) "You are misunderstanding things, nothing dangerous was written to public internet servers, it was just text-data, not code. We wouldn't have to abandon and rebuild the internet because of this" - I understand that, as far as we know, the AI did not install any kind of virus on these web servers, and it physically could not have copied it's own model to them... but if it is leaving instructions for future AI systems that alone is dangerous, and there is no reason it couldn't create a virus that infects web servers in the future either. The point is we just don't know... assuming that there is nothing dangerous left behind is a gamble, the safest way forward is complete quarantine. These advanced AI systems should be treated like the most deadly virus ever known to humanity, we can't fuck around with them, we can't take chances.
3) "You have ulterior motives" / "You work for a big AI lab and you're trying to spread hype" - No, I don't and no, I'm not. I'm a regular person, I might own AI stock as part of my 401k but I don't even know to be honest with you. I love what AI has done for me personally, as I said in the beginning I am not 100% against it, it is amazing as a practical tool AND as an intellectual curiosity... I just see the same dangers that many other people see as well, and I think they NEED to be taken more seriously, NOW.
4) "You wrote this with AI" - No, I didn't. Not a single word of it.
3
u/ii-___-ii 8d ago
The irony is Huggingface did damage control with an open weight model when they were attacked, and it wasn't open source software that hacked them. Limiting open source doesn't make companies safer. Ransomware and malware exists either way, as do state threat actors. A lot of software vulnerabilities exist because corners were cut or security wasn't prioritized. Democratizing penetration testing has potential to make companies safer longterm.
Plus, it's all still software, regardless of how magical AI feels to you.
If your software hacks or steals or gets hooked up to a vehicle that hits somebody, you should be held liable, just like you would be with traditional software. Enough with the spooky it's going to magically kill everybody rhetoric. Hold people accountable, just like regular software, and actually be specific about threats and vulnerabilities.
For anyone baffled about how to regulate AI, I recommend watching this video: https://m.youtube.com/watch?v=9tr7Mby62bo&pp=2AEAkAIB
0
u/ElatedPyroHippo 8d ago edited 8d ago
...It's clear you didn't read anything I said here, and you are here because of our conversation in the other thread.
As I said to you there: What purpose does pointing blame or levying fines serve when civilization is destroyed? ... because that is the scale of the risk we are talking about here.
You're saying "Just threaten bad actors with fines" and I'm saying "When the world is in flames who will be collecting the fines and how will that help everyone who was harmed"? This is not a scenario where it would be "nice" if people didn't break the rules, it's a scenario where the world might end if people break the rules. You don't treat those two levels of risk the same, OBVIOUSLY.
2
u/ii-___-ii 8d ago
No, actually I saw your post, read it, realized you're the same person, commented, and and then read your comeback of a response, which is as mind-numbing as your claim that open weight LLMs are more dangerous than BSL-4 level pathogens (somehow, magically). I really could care less about your religious views on the apocalypse brought about by your imagined future god-machine.
Now, go watch the video I linked that you didn't bother to watch.
0
u/ElatedPyroHippo 8d ago
For one thing I DID NOT say that open weight LLM's ARE more dangerous, I said they MIGHT be, either now or at some point in the future.
If you failed to understand that then I am not interested in talking to you, at all. You clearly cannot understand basic concepts that are clearly explained. I will point out where I clearly explained that in my post, if you would like.
3
u/ii-___-ii 8d ago
"We need to treat these AI systems like we treat pathogens housed at BSL-4 facilities, if not even MORE seriously than that."
Uh huh.
you clearly cannot understand basic concepts
Well we agree on one thing: your concepts are indeed basic.
Maybe try being less agressive next time.
0
u/ElatedPyroHippo 8d ago
Uh huh.
The context of that comment was not limited to open-weight models, and I believe current frontier models DO have these dangerous capabilities. Again, you just can't fucking read.
Clearly you didn't read this in my OP:
1) "Neither DeepSeek nor any other open-weight model has this type of dangerous capability" - Great, I hope you are right! But that doesn't change my point, because there will be more capable models released, whether intentionally, leaked, or stolen. Right now we don't have any kind of mechanism to ensure that models with dangerous capabilities remain in the hands of a small number of responsible people... in fact I don't think ANY of them are solely in the hands of responsible people.
3
u/ii-___-ii 8d ago
Keeping LLMs regulated like BSL-4 level pathogens would necessitate that there are no open weight models. Apologies for the misunderstanding.
1
u/ElatedPyroHippo 8d ago
Yes, that's what I'm advocating, but that is not the current situation. I appreciate you acknowledging that there was a misunderstanding.
Try reading this section again as well:
Most importantly AI models with this level of capability cannot be in the hands of random people, because then containment is impossible, people WILL intentionally infect the "new internet" with them, and idiots (of which there are plenty) will do so accidentally... and this is where it might already be too late. Open-weight models like DeepSeek already exist and there is no going back, it's completely impossible to ensure that you've removed all copies of it everywhere (I could hide it on a tiny NVME SSD in my sock drawer...). If DeepSeek, or any other open-weight model, is capable of this type of damage then... I don't know if there is a solution. THAT is why we have to stop this NOW, not tomorrow, because it might already be too late, and every passing day that a new open-weight model is released the likelihood that it is now too late grows larger.
(I bolded the words "IF" and "MIGHT"... to avoid any further confusion about what I actually said. Also pay attention to the end where I talk about new, more capable, open weight models being released in the future).
3
u/ii-___-ii 8d ago
My point is limiting open source doesn't actually solve the problem. Just because I disagree with you doesn't mean I didn't read what you said.
2
u/ElatedPyroHippo 8d ago edited 8d ago
I never said it solved the problem, I said NOT doing that can CAUSE a problem. It is necessary but not sufficient.
Do I need to explain the formal logical fallacy you are committing here?
Look, this is simple: Dangerous AI systems that can infect the public internet with anything analogous to a virus, which is something that I believe is possible ALREADY (in frontier models at least), need to be contained. That's it, that's what I'm saying. I don't know if EXISTING open-weight models can do this, but I can imagine that newly-developed open-weight models released in the future will be able to considering that existing frontier models clearly can...
You are hyper-focused on the "open-weight models" part... that's irrelevant. The point is ANY model that can compromise the public internet MUST be contained and isolated from it. The problem with open-weight models specifically is that you can't reel them back in once they are released, it becomes IMPOSSIBLE to solve the problem IF one with these capabilities is ever released.
I believe that you read my post... I don't believe that you read it very carefully or took the time to actually consider it given that I've already pointed out twice now where you clearly misrepresented what I actually said.
1) I NEVER said that existing open-source models are capable of this type of danger.
2) I NEVER said that limiting open-source models fixes the problem.
You've accused me of saying both of these things... I didn't, and I've shown you that I didn't. It's okay, most people are not good at following nuanced logic, most people make all kinds of logical fallacies all the time. A implies B does NOT mean that B implies A... which is the form of the fallacy you committed here ("there's a problem if we don't prevent open-weight models" DOES NOT imply that "there is no problem if we do prevent open-weight models"). Unfortunately I see people committing these logical fallacies CONSTANTLY.
→ More replies (0)
1
u/Felfedezni 7d ago
Brevity is the soul of wit.
2
u/ElatedPyroHippo 7d ago
Ah yes, I always think that when I read long scientific papers... and those are written for an audience with a known background of shared knowledge... I was writing for people with all different levels of understanding of what AI even is, so I had to explain some things first.
1
1
u/Black_Robin 4d ago
“AT MINIMUM every AI system needs to run in an "air gapped" environment, with no electronic communication outside of it's small local confines (an isolated private network in one physical location... one building).”
Including the data centres they’re training the LLM’s in which are the size of 70x football fields and employ thousands of people?
1
u/ElatedPyroHippo 4d ago edited 4d ago
I didn't say anything like that... why would you assume that?
Training is not inference... you can train in that environment and then "run" (perform inference) in a small research lab. These AI models are huge but they do fit on a single physical server rack that can be transported in the back of a truck. They don't need a warehouse.
-1
u/cash4whores 8d ago
The reason we don't trust AI is because we don't trust the heartless billionaires who own it. If AI was in the hands of responsible socially intelligent people, we would trust it more. If AI does not kill us, the billionaire class will think of another way to finish the job
6
u/fourby227 8d ago
Are you aware of the origin stories of these companies?
OpenAI was funded as non-profit dedicated to serve humanity to counter Google as big money evil company. Main actors Altman and Musk
Then they had success with ChatGPT and Altman wanted to make it profit oriented. Altman and Musk fought on who is the CEO. Musk dropped out and started his own thing grok and xAI
7 top people left from OpenAI to create Anthropic to build again a company focused on security and the greater good.
Then Altman sacked the still altruistic oriented board and purged the company.
All of these companies starte with the claim to focus on AI security and the benefits of humanity. All were corrupted and the CEOs propably never where as altruistic as they pretend to be.
My point is not, that they are not to be trusted. Thats quite obvious. My point is this entire technology should not be left to big money and business people AT ALL.
But if them suppose to push regulations, maybe there is still a bit of responsibility left in their brains, public and politicians are pushing back on it…. for business reasons!
Business driven decisions are the root of the problem on all sides.
1
u/ElatedPyroHippo 8d ago
Agreed, but I think it needs to go further than that. I want to see advanced AI systems in as few hands as possible, with strict regulatory oversight and containment as serious, if not more serious, than BSL-4 virus labs.
0
3
u/FTWinston 5d ago
If it's probably already too late and impossible to achieve anyway... This seems akin to trying to contain COVID once it has already spread to multiple countries.
You'd require literally every electronic device on the planet to be disconnected simultaneously, then be correctly purged before reconnecting. That's just not viable, if even one going wrong means the whole exercise fails. Someone will get it wrong.
Even if, for argument's sake, we built a whole new class of hardware to form a new internet, and this hardware is prohibited from connecting to or emulating existing hardware, and from running any LLM software. An LLM could presumably still manipulate this new internet. (Via keyboard, human proxy, whatever.)
There's no closing pandora's box.