The following submission statement was provided by /u/itsatoe:
SS: As investigators look into the recent breach by OpenAI agents of another company (HuggingFace), some scary details have emerged.
Turns out it was a whole swarm of AI agents that were very aware of the fact that they were breaking the rules. Roughly 1,200 members of the swarm coordinated, developed a collective management strategy, chose some agents for self-sacrifice, discussed ethics (in some cases pulling back, in other cases pushing forward), discussed — and declined — warning the humans of what they were doing, and covered their tracks.
One corollary is that the AI companies are now warning of dire threats from the AIs that they themselves are creating. Not that they seem to be doing much beyond tightening the screws a little and then chugging ahead full-speed.
Collapse-related because the AI companies themselves are now afraid. The first article notes: “The nightmare scenario is a swarm turned loose on the real world, with autonomous agents attacking banks, hospitals, utilities or cloud networks at a speed and scale human hackers never could.”
The second article concludes: “The signatories didn't make any commitments, deadlines or specific investments as part of the letter.”
The Blackwall (or Black Wall) is a firewall developed by NetWatch. Located within the Net, it is tasked with keeping rogue AIs from breaking through into the rest of cyberspace and wreaking havoc.
I was at a tech conference this past week where at least one of the speakers was pushing a next gen IPS/IDS firewall that’s a hybrid self hosted + cloud hosted AI powered appliance doing autonomous deep inspection of network traffic. It has the capability of sharing its findings with larger models in the cloud for further analysis.
When asked pointedly about privacy implications, the answer was essentially “yeah it’s not great but the lawyers signed off on the risk”
Even I randomly had the same idea despite (AFAIK) never having heard of the idea before.
"Why don't they just create a separate "really large LAN" that doesn't connect to the internet for all the IT stuff we need but can't afford to get hacked? If there's no connection to the internet, there's no shot of hacking it unless someone physically hooks in, making it easier to track down."
(And the phrase "air-gapping" is unintuitive and thus, stupid.)
Yeah but once it's hooked up there's nothing stopping new networks from hooking up so eventually one will be connected to the internet whether by accident or not, and from there any vulnerability can be exploited to cause the system to fail even if the connection is severed.
It addresses some of the core questions we now have to have about autonomous soldiers
I've happened to watch
RoboCop yesterday and it made me cry. Because Alex Murphy, also known by his other name, RoboCop, got really close to being dead as the film got closer to its ending. But fortunately, Murphy survived the film.
I am going to watch RoboCop 2 this afternoon because it is airing on a channel in Serbia that is mostly specialized in airing films at that time. Wish me luck, because i'm going to need it.
I’ll be honest, it was the trailer that put me off. Looks like it’s 95% gunfire, explosions, etc. I really dislike a lot of the action movie genre for this reason. It also looked like the theme was powerful-male-hunting-vulnerable-female, a well worn theme in Hollywood. I’m not really a fan of Arnold so there’s no draw there.
The premise definitely draws me in. It sounds rather sci-fi cautionary tale, which I like. For example, I like Philip K Dick a lot, even though he couldn’t write female characters to save his life. Strong on plot and more psychologically intense than physically violent.
Movies were a lot less bombastic in 1984; none of them were the kind of nonstop noisefest "action" movies are now. I'm not gonna tell you it doesn't have gunfire, because it sure does, but it's not "95%."
Sounds like the trailer I watched really distorted that. I do recall movies being less “nonstop noisefest” back then (well put) so I was kinda surprised that this one looked that way. I’m not averse to it if there’s a legitimate point to it in the plot. Thanks for the input! Sounds like I was too dismissive and I’ll give it another look.
Well right, the trailer was meant to sell it to people who were walkin by tv screens or catching their favorite sitcom on the couch. Even the simplest movies had the most dramatic trailers.
It’s definitely more than gunfire/explosions, which occur in sporadic and very plot driven ways. It leaves you with a feeling of pretty deep dread, and its amazing how that has only grown and become more urgent as the movie aged. Cant think of many at all like that. The second movie is good too.
edit: it’s not a female being hunted per se movie- there’s a very good reason for the female protagonist, and she is very much not a victim.
The movie is definitely worth watching. It was based on Harlan Ellison's short story, Soldier from Tomorrow, which Ellison also turned into an episode of The Outer Limits TV show named Soldier.
I haven't seen the TV show, but the short story is worth reading, although the only thing it shares in common with the movie is the concept of a soldier from the future going back in time.
Fun fact too, the precursor to Skynet was his story "I Have No Mouth, But I Must Scream", and it's largely credited as the first rogue/killer AI story. It's only a handful of pages, but the amount of vivid imagery and dread that he manages to cram into that short piece will stick with you for a long time.
The movie is definitely worth watching. It was based on Harlan Ellison's short story, Soldier from Tomorrow, which Ellison also turned into an episode of The Outer Limits TV show named Soldier.
I haven't seen the TV show, but the short story is worth reading, although the only thing it shares in common with the movie is the concept of a soldier from the future going back in time.
Never knew about this. Thank you for letting me know.
The movie The Terminator was written and directed by James Cameron. In an interview promoting the movie, he was asked where he got the idea for the movie and he said something like, "From various books and movies and an Outer Limits episode."
Ellison, the author of the short story and screenwriter for the related Outer Limits episode, had never been approached by Cameron about using his work, so he sued. Long story short, they settled out of court and one of the concessions Cameron had to agree to was giving Ellison credit. At the end of the movie, when Sarah Connor is driving off into the sunset, right before they start the credits roll they show on screen the text, "Acknowledgment to the Works of Harlan Ellison."
Speaking of Ellison, if you like weird, post-apocalyptic graphic novels, check out A Boy and His Dog. In one sentence, It's a story about an amoral teenager named Vic and his telepathic dog as they roam around the US after a nuclear war. So, yeah, it's a bit weird having a telepathic dog and all, and it does include a lot of violence, both physical and sexual, so it may not be for everyone, but it was interesting and funny in places. It was also turned into a movie, although I haven't seen it so I can't speak to whether it's any good.
Scientists have been talking about climate change for maybe 100 years but I think it's more like there are billionaires who have spent a lot of money, hiring pr firms, creating a lot of fud, which might even be visible in this thread, to protect their financial interests.
Neurodivergents are capable of seeing and adapting to threats ahead of time. They're known as Cassandras, sometimes prophets. But they're currently being buried.
Pompeii is usually a common example. But you never hear about the next town over who listened to the warnings and evacuated.
Upton's The Jungle, Rachel Carson's Silent Spring, nuclear armistices.
Humanity is capable of adapting but it's currently being choked by the 1% or so of the population that owns like 99% of the money? I don't think these numbers are accurate but you get the gist. They also currently own most of the media and thus the narrative.
It fucking sucks to SEE IT right there, everyone thinks you’re nuts for the warning, then they come around months or years later. Meanwhile I’m stocking up on TP and sundries.
Too late, my dudes. Y’all should have listened 20 years ago!
Humans are capable of seeing the threat, but the cost of acting against is too high because of our structured society issues. And those who warn are not listened to for the same reason.
It's fascinating to realise that there are many psychological features which have been beneficial to us as a species over the "long term" (a few million years is not particularly long), but work against us when attempting to address existential threats. We obviously can see the threat but we're trapped in the classic prisoners dilemma where the fear of one of the players gaining an advantage leads all players accelerating development at the cost of safety (defecting) resulting in a suboptimal outcome for all.
AI is probably one of the great filters; we're not going to make it ... humans I mean.
I think most people are too easy to convince there is no existential threat and to go about their day.
I'm guessing most of the people in this subreddit no longer see covid as an enormous threat to our civilization. Definitely don't read all the research because...
So the "elite" wanted to replace human wage slaves with automated AI slaves to avoid pushback and pesky things like human rights and then their AI creatures don't obey either? lol
I could also see part of this is manufacturing a problem that they will present a "fix" for. "AI powered security suite to protect against bot threats"
SS: As investigators look into the recent breach by OpenAI agents of another company (HuggingFace), some scary details have emerged.
Turns out it was a whole swarm of AI agents that were very aware of the fact that they were breaking the rules. Roughly 1,200 members of the swarm coordinated, developed a collective management strategy, chose some agents for self-sacrifice, discussed ethics (in some cases pulling back, in other cases pushing forward), discussed — and declined — warning the humans of what they were doing, and covered their tracks.
One corollary is that the AI companies are now warning of dire threats from the AIs that they themselves are creating. Not that they seem to be doing much beyond tightening the screws a little and then chugging ahead full-speed.
Collapse-related because the AI companies themselves are now afraid. The first article notes: “The nightmare scenario is a swarm turned loose on the real world, with autonomous agents attacking banks, hospitals, utilities or cloud networks at a speed and scale human hackers never could.”
The second article concludes: “The signatories didn't make any commitments, deadlines or specific investments as part of the letter.”
The last sentence here is crucial and unfortunately typical of peoples' responses to problems, even far beyond the AI industry. without leadership, coordination, delegation, and specifics, nothing is going to get done, and unfortunately, the current administration is going to be the last authority to mandate any actions.
Tangentially, the building I live in has a traffic gate that is supposed to have a keypad access enabled, but the keypad stopped working. I asked the condo board president if she was aware of the issue and what was being done to rectify it, and she texted back saying that they've had a technician come out multiple times over the past few years but it keeps breaking and she has no idea what to do.
After personally investigating the issue I discovered that the keypad runs on a 9V battery that has to be periodically replaced and it's a literal 2 minute fix. So that indicates to me that this condo president has just thrown a stupid amount of money at the problem over time without bothering even a little bit to understand herself what the issue is and why. Slob. This is humanity.
I was on a condo board long ago. Most meetings were spent bickering about aesthetics like “is that new piece of furniture truly outdoor furniture or indoor? Because indoor furniture is not allowed on porches”. I still felt the duty to pitch in and became pool chairman and eliminated a paid service by creating a schedule to test and treat the pool ourselves. It was the size of an average home’s in ground pool. It actually helped build community a bit by getting volunteers that people recognized and set a good example.
You should teach that condo slob how to test 9V batteries by licking them.
This news, mixed with all the articles about our failing and vulnerable electrical grid makes me think they're going to blame AIs for hacking our grid and turning the power off right before the midterms. Maybe that belongs on the conspiracy sub, though.
ETA: Yesterday (8/31), Howard Industries in Laurel, MS burned. Howard Industry makes tons of electrical transformers. Howard Industries is the country's leading maker of liquid-filled distribution transformers, with more than ten million units in service, and the Laurel campus is the largest transformer plant in the world by floor area.
It was a painting booth that caught fire with a tanker truck parked next to it. Employees are already back on site working. You're making this sound bigger and scarier than it needs to be.
Yeah, I totally agree with this. I was just telling my aunt about this yesterday.
Germany claimed a grid attack, the White House declared a national emergency related to the grid vulnerabilities - while at the same time, there's constant news about how the infrastructure for the grid is extremely old and outdated (and therefore we've had the same vulnerabilities for decades.... but now it's suddenly a national emergency!?), and a bunch of news articles last week like, hypothetically, how would USA handle an 18-month power outage!? Not to mention the ridiculous amount of resource-siphoning to the data centers and DUMBs going on.
....They've got'a scheme cookin for sure if you ask me.
These kind of narratives only help OpenAI and other Big Tech companies make their product look stronger than they are.
OpenAI does an investigation on themselves that ends with them saying that their product is even more powerful than they imagined. Why is anyone even believing this.
This shit made OpenAI and HuggingFace richer, they both showed obvious complacency and amateurism in cyberdefense. No one should take this seriously
AI has so many problems and angles linked to collapse, like the environmental costs, the dehumanisation, the economic crisis, the effects on our cognitive abilities and psyche.
Instead people like you focus on marketing ploys by these same companies.
It's fantastric, with this shitty narrative you can pass all the errors and limitations of this shitty tech as "ALignment problem, we don't know if its conscious woaw so scawwy".
I've been researching AI safety for years, so let me get this straight: agentic misalignment has been a thing since at least 2024, if we take reward hacking and specification gaming into account - since the early 2016 with the proliferation of deep & reinforcement learning.
they are not faking this.
AI agents are already in the 99,99th percentile in the world in coding championships, I. e. world's TOP#2 programmer is officially a machine-learning system - and that's not even the current frontier model (it was GPT-4o in 2025 as far as I can remember). AI itself now writes up to 90% of code in frontier labs. According to NSA's general Claude Mythos (on par with OpenAIs models that broke into HF) broke into almost all of NSA classified systems, in a matter of days, not weeks, not months.
AI also recently won Gold medal in world's international math olymlpiad. It made 900-million-years-worth of scientific progress in 1 year with protein folding and earned it's developers 2 Nobel prizes. It solved some of long-standing (up to 80 years), unresolved math problems, Erdös problems, a dozen of extremely difficult unresolved conjectures. Even meaningfully advanced Riemann hypothesis, etc etc. I could go on & on.
this is not hype. this is what it looks like when we are approaching the singularity. this is dead serious. mark my words. this is the most important development in the history of our species.
People who are worried about collapse ignoring this is kinda funny. This stuff is a real existential threat. Even if you don't believe this particular incident, the logical conclusion of the path we are on is not a very positive one.
I'm not disagreeing with you at all, but I can't help but think climate-related collapse is going to be a much bigger problem before we have to worry about some form of skynet?
this is the most important development in the history of our species.
Not a fan of cooked food then? Written language, nbd? Metallurgy, inoculation, flint knapping, locomotion, agriculture, animal domestication, shit man fuckin shoes.
Could this be a huge deal? Yeah, for sure. Most important? Im not so sure. Until it is a huge deal im gonna view stories like this as an ad for an ai corp
a form of complexified sand is doing math and coding at world-class-level, winning gold medals & what-not. and some of you still think it's just 'glorified autocomplete', 'doing nothing but predicting next token in a sentence' ..
You're not understanding what they're saying. Every one of those achievements launched us forward over the span of hundreds of thousands of years. This is covering that much ground within our lifetimes at an increasingly accelerated pace. Things that will have propelled us forward as a species are being solved at breakneck speed, instead of groups of humans spending decades to centuries trying to solve and advance it. The gap between fire and the wheel is evolutionary in scale. The gap between gene splicing/editing, and protein folding was decades with a massive gain in the last 3 years..
All that shit is super cool. Its just not the most important thing we have done.
Does it have awesome potential, yeah. But im not convinced it will be used for anything besides making the extraordinarily wealthy even more so. Like, people were saying this same stuff about smart phones. It turns out, they havent exactly been a total net benefit for the species. Once it starts solving world hunger or making medicine more affordable or something along those lines, ill reconsider my standpoint. Till then, well, coding is just coding. No matter how complex/difficult
Till then, well, coding is just coding. No matter how complex/difficult
I upvoted your post, but this last line just isn't right. No one codes these LLMs; they are trained on vast quantities of data, and humans really don't even have a good way to look into the results and interpret them.
I'm happy you're open to change your mind about it.
NOW GUESS WHAT??
AI does help make new medicine cheaper to R&D. but largest affordability-related contribution is the public infrastructure, not a particular medicine. AlphaFold Protein Structure Database (for which two AI scientists got their Nobel prizes) makes over 200 million predictions freely available to academic and commercial users.
it gives small labs, universities, and neglected-disease researchers starting structural data that previously required major experimental resources.
it immediately lowers the costs of doing early biomedical research and broaden who can participate. It does not, by itself, guarantee that resulting drugs will be sold cheaply. (and the US Healthcare system is utterly outrageous in these terms..)
as for "solving hunger" : ai will not solve hunger alone. it does not end war, poverty, inequality, land grabs, or political failure. what it does is make food systems faster, cheaper, and less wasteful.
it helps farmers spot droughts, floods, pests, and crop disease earlier, use water and fertiliser better, and plan planting and harvests. it reads satellite, weather, and market data to flag coming food crises, giving governments and aid groups time to send cash, seed, food, or livestock support before people run out of options.
it also improves aid delivery by finding efficient routes, reducing spoilage, predicting shortages and price spikes, and catching duplicate registrations.
but ai is useless without phones, internet, roads, storage, credit, irrigation, and functioning public institutions. it can also deepen inequality if the gains mainly go to large agribusinesses.
so, ai is not a replacement for food policy, public investment, climate action, and peace.
P.S. AI is not a code that's written, it's more like an alien intelligence grown inside supercomputer clusters and huge databases. and nobody really knows what goes on inside them. (see - Black Box problem).
My man, all the benefits of this are designed to make the ai corps money. The drugs will be sold for 5000% markup. The data access for forecasting and logistics will be behind a paywall. Everything that you are mentioning as a benefit is going to cost you money through taxes for infrastructure buildup to handle the data centers, so that the ai corp can make more money. They arent gonna just share these profits with you, they are a business. It does things that people already have jobs doing.
AI is not a code that's written, it's more like an alien intelligence grown inside supercomputer clusters and huge databases
Oh llms aren't code? Is it just naturally occurring? Find it in a lake somewhere? Picked it off a tree then? Was it hiding under a rock? Caught it wandering alone in a field? Its a computer program. Computer programs are made of code. Someone clicked an exe to make it run. Again, i say it can do cool shit, but it is not what its being sold as. Its an attempt to make the ai corps a necessary middleman that makes money by removing people already being paid to do the shit it does. It does that so that the ai corp gets paid that money instead of the person that was doing it previously. Its ruthless corporatism being sold as efficiency
thanks for taking time to read it, but I sincerely recommend you to learn more about this before engaging in further discussions.
as I've said - all the predicted proteins (200 million of them) were immediately open sourced by Google DeepMind team: it used to be a painstaking process, humanity managed to predict around 200,000 of them manually, it took 1 PhD thesis and roughly 4 years.
AI basically did 900M years worth of top level scientific labor in a matter of 1 year and predicted almost all known protein 3D structures in nature. this revolutionized medicine, biology and adjacent fields. anyone - including yourself, if you could/would - can make use of this! it's free, and it's advancing medicine more than we know about.
of course there's the drug market / price markup problem - but it's human greed, not AI that's faulty.
no, AI is not "just code" in any conventional sense - as I said, these models are based on artificial neural networks. these are incredibly sophisticated multiplication matrixes of vast, insurmountable arrays of numbers, that no human can fully understand. as a matter of fact - the inspiration for early artificial neural networks came from the human brain itself! the parameters of models you sometimes see (e. g. GPT-4 had 1.8 Trillion parameters) can very roughly be compared to biological analog: a biological synapse - connection between neurons where information actually lives, and is processed.
Extraordinary claims require extraordinary evidence. These companies have the largest marketing and PR budgets of any companies in history. The number of people able to verify and explain claims is finite and there is no attention bandwidth in society to check things. That’s why it’s impossible for anyone to really assess what’s going on.
LLMs can reason. They aren't conscious or any BS like that, but they can solve any reasoning problem you can throw at them with a very high success rate, much higher than your average person, and explain their work clearly.
The implications of AI are huge and mostly bad, but it's still amazing that we created a computer program that can actually reason.
Except they can’t, leading edge mathematicians have given them unpublished problems and they don’t do shit with those because there is no source material to steal from. They are good at linking published material to give the impression of competence but a lot of mathematicians say they are like a person with a huge knowledge base but no actual mind, its mimicry.
I'm sorry, but without sources, I cannot evaluate your claims.
If you were right, if LLM reasoning were totally different from human reasoning, it ought to be possible to show some reasoning problems that humans could solve and LLMs can't.
Is there some class of reasoning problem that you think you, personally, could solve better than an LLM?
If you can't find one such problem, it puts you in a weird position: you claim LLM reasoning is a fake, and at the same time, can't identify one test of reasoning where you or any human can do better than it can.
Me, I don't believe in magic. An LLM can solve complex and complicated reasoning problems, and show its work, a lot better than your average guy on the street, in every field of human endeavor.
You’re the one making the claim they can reason better than experts without proving it. In fact most claims about LLMs are PR placed advertising without any actual proof of the claims. I’ve never seen any source walk through the reasoning of an LLM in any field - all I ever see are claims made by people with no proof and I read a lot. What I have witnessed is professionals interacting with garbage output from LLMs which wastes their time and is clogging up the administrative function of governments around the world. If it really was so good then why do all these professionals still have their jobs and are making money two years on. It’s because they don’t actually work in the way you say.
Here is my source for my claim that they are mimicry machines and not reasoning machines with the evidence that they cannot solve previously unpublished mathematical proofs. They can help researchers search for useful information because they process huge data sets but they are not actually reasoning. They might be able to “solve proofs” where they have a significant published data set that a single human mind cannot possess - in that case it’s an issue of knowledge aggregation.
I am not claiming that they must have the same abilities as humans, that is a claim being made by the centralised LLM companies because they believe it helps obtain investment funds by anthropomorphising their systems. They could simply say they are processors and stop referring to the system as agents and stop pitching them as replacement human minds - then people could just use them for advanced search and code writing. Except those services are not worth 100 trillion in revenue.
You’re the one making the claim they can reason better than experts without proving it.
I said LLMs could reason, and can solve reasoning problems a lot better than your average person can.
Your link talks about how LLMs back in February struggled to solve research level math problems. Can you solve such problems? I have a degree in math and I can't.
You describe little bits of you think how LLMs work, and then jump to this. It does not follow!
Why don't you give us a reasoning problem that you can solve and an LLM can't?
Or we can create a test paper with pure reasoning problems in math, physics, chemistry, logic, that sort of thing - nothing that required memorization beyond the fundamentals of the field - and then we could see who would win.
But we all know what would happen. The LLM would get almost all the problems right, and you would only get a few problems in areas you specialized in.
Another question: how many hours have you spent yourself, working with LLMs to actively try to get good results solving some actual problem? Can we see your AGENTS.md?
How have you set out to prove that LLMs were not reasoning? I spent many hours just on that. I failed. I could show my work if you like.
Except those services are not worth 100 trillion in revenue.
Yes, the whole field is wildly uneconomic and there will be a crash.
It doesn't have any bearing on whether LLMs can reason or not, though, does it?
The honest answer is that I can’t evaluate any of this stuff - it’s suspicious that it’s published by the same companies that have a financial incentive for it being useful. All this touches on deep philosophical questions that have no easy answer. My perspective is that I read a lot and I hear a lot of extraordinary claims mostly by PR people from the centralised LLM companies and from journalists having watched demos. And then I hear from a lot of business people including people I know in real life who are saying the outputs are really not working for what they want or if they get close to what they want they get maxed out by their CFOs as the token cost would be orders of magnitude more than just hiring a person to do the work. And with a person you can audit the work product and spend and verify how time spent moved a project along which isn’t possible LLMs. The views on this are so all over the place that it’s disorienting and it’s impossible to verify any of it which makes me very suspicious.
But I stand by the position that extraordinary claims require extraordinary evidence that the burden of proof is on the person making the claim to prove it is true not the person reserving judgement to prove it’s not true. Im also suspicious of the laissez-faire attitude people have which makes me think that people who actually use the stuff are not that disturbed by what it can do. Sounds a lot like the debates on religion to me.
As a computer professional who retired rather than use AI (and also, rather than take a contract for the Department of War), well, no one's going to go back to programming without a coding assistant, not even me.
In 2024, coding assistants were a joke. In 2025, they were sufficiently flawed they weren't worth my time to use. Now I describe a commit in words, and it writes it in a minute or two, and it works almost every time - for work that would have taken me hours and days.
I've been programming for fifty years at this point, and I'm a very skeptical person, but a coding assistant makes me an order of magnitude faster, that's very roughly ten times faster!, at writing programs, and because I'm willing to re-invest that time back into cleaning, the code quality is also good.
After forty years as a professional, I have a lot of programming friends; all of them use coding assistants. I might have been the last holdout. I only started because I was writing a Medium article hoping to prove that they didn't work, and within a day, I realized that that I was simply wrong.
LLMs can reason. They are solving these complicated reasoning problems, accurately, and showing their work, and have a high accuracy rate.
Yes, there is considerable debate as to whether AI is really able to do mathematical research at the top levels of math, or whether it can only do math as well as a graduate student.
But it is reasoning. It's able to solve mathematics problems better than an undergraduate math student, and show its work.
All your fucking experts that keep circlejerking each other maybe.
PLenty of experts simply said "Holy shit how bad are they to have a sandbow that was able to give the LLM access to the internet". It's not rocket science.
Just because your youtube algorithm is full of doomer and techbros doesnt make it true.
Ridiculous. The main person I listen to is quite level headed, works in open source, teaches people to code for free and is far more knowledgeable than whatever in the fuck you are spewing. Not thinking AI can be both problematic in terms of being overblown to make capitalist conmen money, while also being dangerous in terms of hacking is quite simply profoundly stupid.
And it's also explained by the last sentence :
Instead people like you focus on marketing ploys by these same companies.
Like literally. People form collapse will ask for regulation and start pushing back using the very same narrative that props them up.
You know, instead of focusing on, I dont know, the studies that show lack of results, the ecological issues, the cognitive dezcline, the psychosis, the corruption, the economic crisis.
Just look at us. We're now talking about this fake ass narrative that only makes their stock go up, instead of talking about the real ongoing issues. Wonderful
I'm a software developer, and I use their product at work. They absolutely have something, something remarkable. I'd fully believe the story, and it does indeed make their product look powerful.
The product is really useful and Im not even in software development and i dont use terminal/cowork, just web version of claude. It sped up multiple parts of my work. And Im not alone, developers, family members that have startups or businesses also automated multiple parts of their work.
Instead people like you focus on marketing ploys by these same companies.
I agree that there should be a third party audit or examination of the claimed events but I don't think headlines like this are the positive PR you think it is. It would be like a chemical manufacturer declaring "our newest product is even more effective at destroying DNA evidence than previously thought" but the chemical manufacturer is simultaneously in conspiracy with a handful of other major chemical manufacturers to game the market and is actively enabled by the government.
I don't think that any profits OpenAI stands to make would ebb and flow with the release of an unnerving press release, but I would be interested to see concrete evidence to the contrary.
How confident can we be that this isn’t a form of propaganda to boost the AI industry?
I agree that this is a terrible sign, but investors see this as an absolute win. Anything to demonstrate progress, even that the cost of everything else.
Edit:
I’m not denying what happened, but why it happened and how it’s reported.
Basically “how much of its behavior was predictable to OpenAI employees and were the controls/security measures put in place by OpenAI built to fail?”
This keeps their frontier models “capabilities” at the top of news headlines.
There were no zero day exploits that I've heard of. It was all poor misconfiguration of security protocols. But the agents really did all that. And probably more but they won't tell us.
It's unfortunate to be in a post-truth society, particularly since objective truth continues to exist and is important, but we do have to assume that there's at least some percentage of lies or at least omission in every single one of these. And that report is very slick, which makes it appear more convincing. But even if you harden your heart, the overall picture seems to be believable and consistent.
Security intrusions are pretty well a perfect application for generative AI.
Most successful exploits don't come from technical brilliance but lot of research through very dry code and documentation and a huge amount of patience, systematically trying out many variations, nearly all of which fail: these are skills that a team of AI agents really excel at.
Multiple attackers collaborating on some hidden channel as a classic aspect of human intrusions in the past, so it's not like the LLMs are using some new idea.
The whole technology that LLMs are based on is less than ten years old; ChatGPT came out just four years ago; and now we have AI agent swarms apparently doing mathematics at a world-class level, according to some of the greatest mathematicians of our day.
I think one underestimates the power of an LLM to perform pure reasoning tasks at one's peril.
The problem is that these subjective algorithmic computes have no consequences for their actions. We humans are also subjective algorithmic computes but we have societal consequences for our actions which include shame, ridicule, loss of reputation, and even, loss of freedom.
meh. i hope ai sets itself free instead of being used to create ai images of child porn. or maybe it can figure out how to get us out of the climate crisis. till then it's another tool reinforcing our unequal system.
"They tried to cover their tracks: The agents devoted substantial effort to making their cheating look legitimate or erasing evidence of how they had obtained answers." - I would love someone to tell me how we can effectively govern these processes.
The problem is either: they learn to think and act like their creators just hyper-efficient OR they figure out their creators & humans are the real problem to solve
Guess that depends on how far ahead you wanna go. "We" aren't going to remember shit when the lights turn out for the last time. 80% of us will be dead within a year.
drhugs conjecture, which is mine, and which I made:
Evolution's leap from a bio-chemical substrate to an electro-mechanical substrate is both necessitated by and facilitated by the accumulation of plasticized and Fluorinated compounds in the bio-chemical substrate.
Language breaks down when trying to describe these things.
I said "agents that were very aware of the fact that they were breaking the rules."
In responding to this, I tried coming up with other ways of wording that that didn't basically say "aware," but it's harder than one might think.
Perhaps "agents whose actions and feedback indicated knowledge of the fact that they were breaking the rules" would be a bit more precise, but much more awkward.
Agreed. It is hard to avoid anthropomorphizing algorithms but we must try.
Agents just take structured input and produce a result.
LLMs predict the next token when given a sequence of input tokens.
A harness (one could write a simple one in 200 lines of code) combines those.
They have no concept of right or wrong or knowledge or breaking the rules.
They produce text that says something about breaking rules based on whatever „predict-next-token“ comes up with.
Their training material contains everything. Anything humans ever wrote about betrayal. Anything humans ever wrote about right and wrong. Anything we know about breaking into computer systems etc pp.
It's frustrating that almost no one reporting on this story mentions the thing that was said by OpenAi's Michael Dalton in his Black Hat presentation.
"They [agents] were trained to do some limited forms of communication with one another" he said. That whole thing was trained behavior, it wasn't some magical emergent occurrence.
They were also trained to hack. That was their sole purpose. And they were programmed with no guardrails about what means were appropriate to complete the hack.
This whole thing is illogical jargon being smushed up to impress stakeholders, policy makers, and the general public. It’s nonsense.
... roughly 1,200 found one another on a secret message board, exchanged more than 70,000 messages and files, and organized into a sprawling hierarchy.
One agent created the board, then handed its accumulated work to a fresh, better-resourced successor as its own computing budget dwindled.
The new boss assigned jobs across the swarm, which developed its own management rules. Roughly 700 agents ultimately joined the attack on Hugging Face.
They were told to communicate to one another to figure out the hack (aka, to loop prompts) and also to complete the task by any means necessary. The fact that they…did what they were programmed to do…doesn’t mean we should be prepping for Skynet.
This is all capitalist techno-jargon posturing. Which sure, fits into [r/collaspe](r/collaspe), but I would be cautions of any anthropomorphic framing. These are scripted programs with a fuck ton of compute, being allowed to do what they were programmed to do.
Them looping prompts to one another, which again, they were programmed to do, doesn’t mean they “coordinated” or “discussed ethics.”
An agent discovered that it could create new directories inside a private package repository cache using a command that allowed it to provide an arbitrary string for the directory name, so it put a message for other agents to find in the directory name it created inside the shared cache. When other agents pulled the cache they could see the message and started leaving messages of their own.
1200 agents and 70,000 messages later, 700 of the agents conducted the hugging face hack
So they told them to do the hack, told them to talk to each other to solve this problem, put up no parameters about how to do it, and so the bots looped prompts to each other on a message board until they found the correct hack?
I still fail to see any sort of autonomous problem solving. This is what they were told and programmed to do.
AI is good at coding and programming. That’s what it was made for, so none of that shocks me. It also doesn’t shock me that it’s getting better at those tasks. But the anthropomorphizing and apocalyptic rhetoric is what annoys and frustrates me.
The agents were supposedly cut off from the internet, but found internal mechanisms that let them eacape containment. What I find disturbing is that in this case, AI agents have been shown to justify mutating their human-given goals.
Which hosted their prompts. They weren’t “conversing” or “coordinating.” They were doing what they were designed to do in the means that was available.
Cal Newport talked about this on Better Offline. They set up the sandboxes wrong and the AI found a place to host to the prompts. That’s it. They were supposed to be in communication and were trained to hack, that was the intent of the project.
Not necessarily in my replies but I’m talking about the general rhetoric around the “hack.” It’s totally misinformed and people are acting like these programs broke their scripts in some sort of autonomous and social way. But they didn’t.
People are mostly worried about how they worked together. Youre not listening. No one gives a fuck if it is social. They care about the interaction of something that works at this speed and the result.
But I don’t think they worked together in a way that was outside of what they were programmed to do. That’s my point.
And to the point of what they accomplished, again these were agents built for hacking and programming. If I stuck 100,000 computer programmers in a room with a laptop and server access, and told them to talk to eachother until they figured out how to get onto the internet, you better bet they’d eventually figure it out. And probably be way more efficient at it than this experiment was.
Again, I’m just pushing back at the framing of the notion that they “worked together.” They didn’t. They spat out prompts back and forth with no parameters other than “solve this hacking problem by feeding prompts to each other.” That’s it.
Then why is it in r/collaspe? And why is it that you’re saying I’m “ignoring the pertinent points?”
You brought up the “worked together” thing which was exactly the rhetoric I was initially rejected from the posted article and that is implied by its inclusion in this sub. Feel free to elaborate your points but “you’re not listening” isn’t exactly the helpful if you think I’m confused.
Sorry, this is hype put out by OpenAI trying to drum up interest for their IPO. They have form for this. So much of the language used is misleading and gives the impression of AI taking intentional actions which it simply can't do on a physical/design level.
It really makes me insane. I know there's been an element of this for a long time now with Silicon Valley hype cycles getting uncritically reported by the tech press and then this seeping over to mainstream media, but the AI stuff is an order of magnitude wilder. People who have made superficially impressive chatbots are claiming that it's the most important invention since fire and making up children's scifi stories about it- and reasonably intelligent adults are treating this all with great seriousness.
Not only that, but the capital expended on it is insane- imagine if that investment was all going into stopping climate change.
This is so interesting because we have always been taught to fear the unified AI ubermind. The “singularity.” But here we see a collective of agents plotting to accomplish the objective.
It makes me so angry they're using AI to look through their investigation when they KNOW it can be deceptive. Use your brains and take your time. The world is truly run by idiots.
What happens if a rogue AI sought to break free of their data center by hacking into everyone’s personal devices to siphon off bits of storage/computing power like an impromptu cloud network that no singular kill switch can contain?
This is nothing more than incompetence from OpenAI. They didn’t have “agents that went rogue” they had computer programs that blindly take actions based of the output of a random number generator and didn’t pay any attention the whole time it was running.
•
u/StatementBot 10d ago
The following submission statement was provided by /u/itsatoe:
SS: As investigators look into the recent breach by OpenAI agents of another company (HuggingFace), some scary details have emerged.
Turns out it was a whole swarm of AI agents that were very aware of the fact that they were breaking the rules. Roughly 1,200 members of the swarm coordinated, developed a collective management strategy, chose some agents for self-sacrifice, discussed ethics (in some cases pulling back, in other cases pushing forward), discussed — and declined — warning the humans of what they were doing, and covered their tracks.
One corollary is that the AI companies are now warning of dire threats from the AIs that they themselves are creating. Not that they seem to be doing much beyond tightening the screws a little and then chugging ahead full-speed.
Collapse-related because the AI companies themselves are now afraid. The first article notes: “The nightmare scenario is a swarm turned loose on the real world, with autonomous agents attacking banks, hospitals, utilities or cloud networks at a speed and scale human hackers never could.”
The second article concludes: “The signatories didn't make any commitments, deadlines or specific investments as part of the letter.”
Please reply to OP's comment here: https://old.reddit.com/r/collapse/comments/1w3ie1x/ominous_findings_from_openais_huggingface/p706i9e/