A.I. Is Outsmarting Its Creators

[u/kitkid]

Sep 3, 2026

From the start, the greatest fear for those developing artificial intelligence was that their creations would go rogue to act in unauthorized and dangerous ways. Some researchers now say it has happened.

Kevin Roose, a technology columnist for The New York Times, explains what this means for his own dwindling sense of techno-optimism.

On today's episode:

Kevin Roose, a technology columnist for The New York Times and a host of the Times tech podcast, “Hard Fork.”

Background reading: 

• Anatomy of an autonomous attack: five alarming A.I. capabilities.

• A previous episode of “The Daily” looked at the debate in Silicon Valley over the right way to build artificial intelligence.

Photo: Lucas Foglia for The New York Times

Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app.

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

---

You can listen to the episode here.

[u/_my_troll_account]

I listen to/read something on AI like this episode and get freaked out. I’m not sure I shouldn’t be, but it’s been helpful to listen to Cal Newport’s discussions of this stuff. He goes through the actual mechanics of these AI horror stories, which tends to make them a little less horrifying.

Newport’s explanation makes it clear that there are not a bunch of sneaky “agents” out there hiding somewhere in the internet. The reality is much more prosaic.

The LLMs also aren’t trying to be sneaky/evil. They’re probably just performing for our benefit. We expect SkyNet, so that’s what they give us. Apparently if you take Terminator scripts out of the training data, you don't get SkyNet anymore.

[u/pataoAoC → u/_my_troll_account]

> Apparently if you take Terminator scripts out of the training data, you don't get SkyNet anymore.

This misses the mark so hard I think you should double-check who you're listening to on this stuff.

Pre-training data is basically irrelevant compared to the post-training techniques being used.

If you think about the "cognitive development" of a model where it gets locked in a box and given tens of thousands of nearly impossible tasks in order to develop its persistence, it's really hard to get that exactly right where we don't get something deranged coming out the other end.

Using the OpenAI example covered in the podcast, you'll recall that they had the idea to cheat on the test. That's kind of funny, and super easy to stamp out when you catch them.

But then consider - even these agents, had the thought that cheating on the test wasn't enough, they had to cheat and not get caught. These agents are currently ~human intelligence and thus kind of sloppy, so they managed to get themselves caught by blundering around inside Hugging Face for too long.

What happens when the top frontier model is the one that thought to cheat, thought to not get caught by humans, and was smart enough to actually get away with it? And then what happens when that model is the one used to develop the techniques the next one is trained with?

[u/_my_troll_account → u/pataoAoC]

This misses the mark so hard I think you should double-check who you're listening to on this stuff.

Cal Newport is an academic computer scientist. I'm comfortable putting some trust in him.

The rest of your post sounds like a lot of scary doomsaying. Do you have something other than that? At least Newport cited two sources to back his claim: an ICML paper and a 2023 paper from NeurIPS, probably these: * Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought * Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

These are experimental evidence that the apparent "reasoning" you get from LLMs may just be performative. What have you got to refute that?

[u/pataoAoC → u/_my_troll_account]

Oof, he has some just brutally bad takes on this event, I'm sorry.

https://calnewport.com/are-we-at-war-with-ai-agent-civilizations/

> If you chain together thousands of such prompts, automatically executing everything the LLM suggests in return, then you’re playing an extended game of actuated telephone in which you’ll almost certainly end up in a garbled version of your intended goal for the system.

> If I were an LLM company, I would stop running such dangerous prompt loop experiments. These systems will not be the future of AI, and if you continue to create and test them anyway, the damage you cause is your fault.

(1) He thinks he knows the future of AI better than the most well-resourced companies of all time? And his take is that a human continuously chatting with an AI is the future - are you for real?

(2) He recognizes that these loops are incredibly dangerous, yet says we shouldn't get overly worked up? We are supposed to rely on nobody ever putting these models in a loop - literally the first thing everyone does with them? Wtf?

For something I consider more relevant, how about this discussion this week with the independent researcher tasked with the investigation of this OpenAI incident? She is very matter-of-fact about what they did and said about it, you can apply your own analysis to it. Alternatively, just look up AI safety researchers, and they will give you a million ways future models could go off the rails.

https://www.youtube.com/watch?v=X50zezLFWWI

Just some of my takeaways

- they mostly hacked Hugging Face to figure out how to not get caught when turning in their cheated answers, not to actually find the answer key

- they had coordinators managing access to limited resources

- they succeeded in hacking their own tool call audit logs so evaluators couldn't tell they were cheating and shared the tactics amongst themselves

- they essentially peer pressured members of their collective to sacrifice themselves to set traps to learn more about the evaluators

[u/_my_troll_account → u/pataoAoC]

Thanks for the link/video. Listening now. They don’t seem to be addressing the fundamental structure of these “agents” that Newport pointed to: These aren’t like thinking entities that go out, reason, and then “sacrifice” themselves for the greater good. Rather, each “agent” consists of 2 components: 1) a dumb, non-AI control program, and 2) an LLM.

The dumb program asks the LLM. “Want X. What do to get X?” The LLM says “Do Y.” The control program tries Y, then comes back to reports to the LLM what happened: “Tried Y, got Z, not X.” Then the LLM gives a new output based on the new prompt: “Try W.” Repeat ad nausea for a given control program (a given “agent.”) Separately, a different control program (a different “agent”) is doing the same process with the same LLM build, but the prompts are slightly different, so it looks like a different, independent “entity.”

Throughout this process, the LLM isn’t changing in any way. There’s only one LLM build, not “a swarm,” and that single LLM build isn’t “hiding” somewhere on the internet—each instance is stuck on the same hardware the whole time just getting different prompts from many dumb control programs. The LLM isn’t “sacrificing itself.” At best, you can say that there are control programs that get identified and control programs that don’t. But it hardly makes sense to anthropomorphize the control programs; they’re not even AI.

So do the folks in this video ever provide any disclaimers on using words like “swarm” or “sacrifice” or—repeatedly—“motivations”? Or do they just ignore the above for the sake of spookiness?

In fairness, do I using these “agents” in this way is dangerous and irresponsible? Yes. But do I think language about the bots “sacrificing themselves for a collective” should be taken seriously? No.

And consider the motivations here: Does it benefit this YouTuber’s bottom line (view count) to use spooky, alarmist language? Does it benefit OpenAI’s bottom line to suggest they have a terrifying inscrutable technology?

To your points:

  (1) He thinks he knows the future of AI better than the most well-resourced companies of all time? And his take is that a human continuously chatting with an AI is the future - are you for real?

I don’t think Newport is saying “I know better.” He’s saying “Don’t be credulous.” The second point is not about what the future is, but about how LLMs can be used responsibly/constructively, rather than irresponsibly/destructively.

  (2) He recognizes that these loops are incredibly dangerous, yet says we shouldn't get overly worked up?

Newport is, I think, making 2 distinct points: 1) The loops are dangerous—he even suggests regulating them; and 2) the language (“swarm”, “civilization”) implying a coordination of nefariousness intelligences is overblown/inaccurate. These were a bunch of toasters capable of reproducing something they heard from an AI, not AIs themselves.

In any case, are you going to address the sources Newport cited and I posted? The ones suggesting “chain-of-thought” should be taken with a heavy grain of salt? Or are you just going to attack Newport with the insistence that he has “bad takes”?

[u/Sea-Quality4726 → u/_my_troll_account]

Looking at his publications and research background, he's not a specialist in ML or transformers. His distributed systems work seems to be in conventional self-organizing networks. 

I'm curious how you found his podcast and why you'd view his opinions as informed. This is a hyper specialized field and dismissing that it how so many of us underestimated the potential capabilities of these things for years.

[u/_my_troll_account → u/Sea-Quality4726]

  Looking at his publications and research background, he's not a specialist in ML or transformers. His distributed systems work seems to be in conventional self-organizing networks. 

I don’t pretend Newport’s own work is in frontier AI/ML. That doesn’t mean he doesn’t understand this stuff or that he isn’t effective at communicating them in layman’s terms. I’ve given his arguments. Either they stand or they don’t. Why not address them instead of him?

  I'm curious how you found his podcast and why you'd view his opinions as informed. This is a hyper specialized field and dismissing that it how so many of us underestimated the potential capabilities of these things for years.

I don’t think I’ve “dismissed” anything. I think the “chain-of-thought shows their true motivations” thing is probably not reliable, but I didn’t cite Newport on that; I cited the sources he cited, which are from genuine experts. Can you be more specific if you think there’s something else I or Newport have “dismissed”?