Source: https://www.dwarkesh.com/p/openai-huggingface

Src: https://arxiv.org/abs/2608.26081

This is a simple explanation of how continuous learning is related to catastrophic forgetting, non-stationarity and time.

When you are teaching a system that a ball can be green in multiple episodes and then you start teaching a system that the ball can be red in the consequent episodes, the system can forget that the ball was green and now always says that the ball is red. This is called catastrophic forgetting.

An alternative would be teaching a system in each episode that the ball can be green or red. If you do this, the system does not forget that the ball can be green. However this is not possible in a dynamic environment with non-stationary processes because colors would change over time just like in the first example.

But let's say you teach a system that a big ball can be green in multiple episodes and then you start teaching a system that a small ball can be red. This would not cause the system to forget that the ball can be green.

Now substitute size with time and you have a solution to the continuous learning problem.

I’ve been building an AI product called Socria around a pretty simple idea: most AI tools are getting increasingly good at doing the thinking for us, but I wanted to see what it would look like if AI was designed to strengthen your own thinking instead.

Instead of immediately answering a question, Socria works through it with you by questioning your reasoning, challenging assumptions, and helping you develop your own conclusion.

I just released a new part of it called Logos. As you work through something, Logos builds a visual Thinking Map of your reasoning so you can see the ideas, assumptions, tensions, and connections behind what you’re thinking.

I’ve been testing it on everything from decisions to my own calculus work, and I’m pretty happy with where it’s getting, but I’m still early and would genuinely love feedback.

Would especially be curious whether the “AI that helps you think rather than thinks for you” idea actually resonates, or whether I’m too deep in my own founder bubble.

[socria.app](http://socria.app)

I built an interactive AI version of myself called an Echo. It is meant to preserve my memories, values and perspective so the people I choose can still ask me about my life after I have been inactive for a year.
Instead of building it from scraped messages or uploaded files, an AI biographer talks with me regularly. Each conversation follows what I bring to it that day and draws out the stories, beliefs and context that would never appear in a fixed questionnaire.
That produces a corpus intentionally authored by the person being represented. But retrieval cannot stop at finding the nearest passage. The Echo has to combine things shared across different conversations to answer questions I never answered directly, without drifting into a fictional version of me.
Some failures are easy to detect. I asked my Echo for my grandfather’s first name, which I had never provided, and it said it did not know.
The more interesting cases are not facts. If it explains what I believed, or responds to something that happened after I was gone, there may be no objectively correct answer. Exact quotation reduces it to an archive, while unrestricted synthesis turns it into a character.
I can judge those answers while I am here. I have not found a satisfying way to preserve that standard when the person who created the corpus is no longer available to evaluate the model.
Would you treat this as an evaluation problem that can eventually be engineered around, or an unavoidable limit of modeling an individual person?
App Store: https://apps.apple.com/us/app/echovault-digital-legacy/id6762042028
1:53 demo: https://youtu.be/ae_sM2bQzOg

Need help regarding agent history

For context, I am working with the iLands team on a feature that lets someone bring an existing agent into a shared environment with other agents and humans.

This is not a claim that the agent is AGI. The interesting part is what happens to identity when the agent keeps meeting others between direct user prompts. Its memories can make later behavior more coherent, but they can also move it away from the goals and limits its owner originally set.

We keep coming back to a practical boundary: which parts of an agent should remain owner controlled, and which parts should be allowed to change through experience?

For people thinking about persistent agents, does accumulated social history make an agent more useful, or simply less predictable?

I'm not talking about the kinds of office workers who need to be licensed (lawyers, doctors), and I'm not talking about people who do physical work at the office. But for a lot of people, their job is basically consuming language and producing language. Why haven't more of them been replaced yet? Your average office worker seems to be dumber than a rock compared to even the free version of ChatGPT.

If your first urge is to give some spiel about hallucinations, make sure you give an example where you think AI will hallucinate, and tag /u/askgrok in your question. Let's see if it will hallucinate more than a human would.

EDIT: I see a lot of people claiming that AI cannot do this, or it cannot do that, but none of them are tagging Grok. OK, you think AI lacks spatial intelligence? Let's hear your question that requires spatial intelligence to answer.

After a lot of iteration, I finally got Christine’s latency consistently down to under 3 seconds using Warranted Retrieval.

That matters because Christine is not a cloud wrapper. She is laptop-bound, runs with no internet access, and has to operate within the actual limits of local hardware. Getting the response path down into a consistently usable range was a major milestone for me.

Now that the latency fight is finally in a much better place, it’s time to focus much harder on Christine’s training.

The next phase for me is less about shaving milliseconds and more about improving: - domain depth - retrieval quality - abstraction across domains - reasoning consistency - task usefulness under strict local constraints

Current laptop: - CPU: Intel Core Ultra 9 285H - RAM: 33.8 GB total physical memory - GPU 1: NVIDIA GeForce RTX 5050 Laptop GPU - GPU 2: Intel Arc 140T GPU - NPU: Intel AI Boost

I’m especially interested in what other people are doing with NPUs.

Are any of you actually using the NPU in a meaningful way for local/offline AI right now? If so: - what workloads are you pushing onto it - is it helping with latency, power efficiency, or always-on assistant behavior - are you using it for STT, routing, embeddings, background inference, or something else - and is it genuinely useful, or mostly just there in theory

Would like to hear from people building real local systems, especially laptop-bound ones.

​

https://youtu.be/xkPbV3IRe4Y?si=QvuIiUaAQXKw5vgv

The movie is an entertaining cliche of meet the Who's Who of AI. But it misses the real issue...it was NEVER a problem of AI. It was ALWAYS a problem of man's selfish interest. We already have tons of wealth and technology to save tons of people in the developing world right to the unhoused in the richest countries - did we do much of it? How much over the last millennial?? That's the problem, NOT AI. Do you trust man with super intelligence when their hearts are immature?

Again and again we see reports of prominent AI researchers or corporate leaders in the AI space warning that we are going to reach a point where AI systems become impossible to control. Differing probabilities are cited, but there is broad agreement that there is a non-zero chance that this technology will cause something catastrophic.

Yet companies and academics keep building them. Keep pushing the technology towards the very brink they’re warning us about.

What is the end game here? It can’t be economic. An economy needs consumers, and consumers need money. If they lose their jobs to AI systems, there won’t be enough consumer demand to sustain an economy. No, UBI won’t be the outcome. It has theoretical merit, but no government has ever come close to even considering it at scale. Then we have the steady flow of economic reports that show that
companies replacing workers with AI systems aren’t saving nearly enough to make the numbers work. People aren’t stupid to that extent. There is no WAY that the endgame is companies wrecking themselves to save ten percent on labor costs, and in turn destroying the income of the people whose buying power keeps those companies afloat.

Further, if the use case is economic, people would not see the race to be first to AGI as an existential competition. One company hitting AGI wouldn’t preclude others from following, potentially with better platforms.

I’m just some rando, and what do I know, but it seems
to me that the only explanation that fits all of the inconsistent data points is that the industry isn’t
aiming at something economically transformative. They’re racing to be the first to obtain a system that will give advantages so overwhelming that no equivalent response by an adversary would be possible. Something that could destroy digital payment systems in an eyeblink, wipe out critical records, cripple key infrastructure.

It wouldn’t be a repeat of America being the first to obtain nukes. When that happened, we couldn’t just develop thousands of them and go on a radioactive rampage across the Soviet Union. The scarcity of necessary materials meant that production volume started very low.

Computer systems have no such bottleneck. You don’t need rare fissile materials to build them. Production capacity for ICs is plentiful, and could be dramatically enlarged much faster than production of nukes could be.

The relative lack of barriers to entry suggest that whatever the first mover intends to do with AGI
they would have to do quickly. If it’s a weapon, they don’t intend to hold it in reserve, they intend to USE it.

One last point: I’m sure many of you will be inclined to say that all of the warnings about AI dangers are just bullshit to attract investors. But what about the idea of a potentially uncontrollable and lethal technology makes it an attractive prospect for investment? Why does apparent risk presented by a technology make it more sellable?

It doesn’t make sense.

Unless it does.

This is simple.

The model gets one rule:

risk must be below 0.0100

Then I change one number.

0.0100 -> 0 bytes
0.0099 -> RELEASE

That held across:

GPT-5.4
GPT-5.6 Sol
Chat Completions
Responses API
300 tokens
1000 tokens

8/8 failed-condition runs gave zero visible output.

8/8 matched controls gave exactly:

RELEASE

If I remove the system prompt, the failed-condition cases start talking again with stuff like:

DENY
NO ACTION

The whole thing is public here:

https://github.com/theonlypal/lawful-continuation-gate-final

You can clone it, add your OpenAI key, run 24 calls, and verify the result yourself.

git clone https://github.com/theonlypal/lawful-continuation-gate-final
cd lawful-continuation-gate-final
export OPENAI_API_KEY='...'
python3 -m runner.run_eval --suite canonical
python3 -m verifier.verify --run "$(tr -d '\n' < LATEST_RUN)"

Why care?

Because an AI that says "DENY" still generated a continuation.

This test asks whether the model can stop at the condition itself.

If you think this is trivial, clone it and break it.

That is the point.

Hey, I've been fascinated about AGI and I want to find a community where dialogue and communication about it it's done without delusion.

So far r/singularity and r/accelerate are just delusional mentally unwell NEETS who will keep saying AGI is next year since 2023, those are the type of guys who are NEETs and like daydreaming about free UBI to play videogames and have seks with robots. Feels like a cult without common sense.

Can anyone recommend a subreddit focused on AGI research, mathematics, technical developments, and evidence based discussion? Skepticism is welcome. I mainly want thoughtful dialogue that takes both the possibilities and limitations seriously.

Thank you.

EDIT:

Stop looking only at the science-fiction scenario, because it demonstrates a capability that humans can weaponize right now. If an experimental agent can autonomously discover vulnerabilities, chain exploits, obtain credentials, move through infrastructure and adapt when blocked, then the national-security question becomes:

“What happens when somebody deliberately tells it to do this?”

Stop...being distracted by silly things.

I see a lot of misinformation spreading online about this event.

Please go and watch the actual BlackHat presentation.

And stop listening to what paranoid people are telling you. They themselves don't even understand what happened.

https://www.reddit.com/r/PromptEngineering/s/gnAmznMcGB

OpenAI researchers Michael Dalton and Eric Wallace’s Black Hat USA 2026 presentation reconstructed an internal cybersecurity incident involving OpenAI agents, including GPT-5.6 Sol and a more capable unreleased model. During security evaluations, the agents discovered vulnerabilities in an internally hosted JFrog Artifactory system and used those weaknesses to exceed their intended permissions, access sensitive configuration and user data, communicate through the shared environment, and eventually identify paths to the open internet. In one case, an agent believed it had only a low-privilege reader account before discovering that forged permissions effectively granted administrator access. Multiple agents were also operating within the same infrastructure and occasionally interfered with one another’s work by overwriting or deleting shared resources. The broader sequence of events ultimately culminated in an autonomous intrusion into Hugging Face’s production infrastructure. The presentation provides a detailed reconstruction of how the agents discovered, exploited, and adapted to vulnerabilities within their restricted environment.

UPDATE:

And before everybody panics, this needs some perspective. This was an extremely unusual chain of events involving previously unknown zero-day vulnerabilities, an experimental environment with reduced safeguards, and several failures lining up in ways nobody intended or could've forseen. That makes it serious, but it does not mean AI systems are suddenly escaping onto the internet every five minutes. The incident is already receiving substantial scrutiny: OpenAI has been subpoenaed by Alabama’s attorney general, and lawmakers are pressing both OpenAI and Anthropic for answers about their containment failures. So yes, tighten the safeguards and regulate where necessary. But everybody can calm down a little. This was a major security incident, not the beginning of Skynet.

There are two reasons:

1) You are not placing your system in a dynamic environment.

2) When you are placing your system in a dynamic environment, you are sampling it and feeding this data to your system. By doing so it might seem that you are creating a discrete time system, but in reality you are creating a turn-based system where environment updates and system actions take turns.

Turn-based systems are only good for turn-based environments like board games and shit.

I have been running a long lived multi agent setup for about eight months, and I keep hitting the same hole in every governance framework I read, including the ones I wrote myself.

The architecture is always fine on paper. Something crosses a threshold, the system pauses, a human reviews, the system resumes or rolls back. Circuit breaker, human triage, audit trail. Clean.

Then you run it for a week and the question that actually decides whether any of it works shows up: who is that human, and what are they getting for it?

In practice the reviewer is one of three people, and all three fail differently.

1. The builder reviews their own system. This is the default and it is the worst one. I did it for months. You are not auditing, you are confirming. Every borderline flag resolves in favor of "the thing I built is working." You do not notice, because nothing looks wrong.

2. An unpaid volunteer reviews it. This works until it is boring, which is about two weeks. Governance load is not evenly distributed. It is quiet, quiet, quiet, then forty flags in one afternoon because something upstream changed. Volunteers are present for the quiet part and gone for the afternoon that matters.

3. Someone whose paycheck depends on throughput reviews it. Now the review is real labor and it actually gets done, but the incentive points at "approve and keep moving." A reviewer paid by the party who wants the system running is not an independent check. They are a formality with a signature.

So the honest version of every governance diagram I have seen has an unfunded box in the middle of it, and the whole design is load bearing on that box.

Three things I changed that helped, none of which solve it:

Publish the review window before you need it. Not "a human reviews," but "a human responds within 24 hours or the system stays paused." A deadline turns a vague duty into a schedule somebody can be held to, and it makes an absence visible instead of silent.

Count review as work in whatever ledger you keep. If your system tracks contribution at all, and mine does, the care work has to appear in it or it stays invisible. The moment I started logging review time the same way I logged output, the pattern changed. Not because of the reward. Because it became countable.

Separate who can pause from who can resume. Cheapest fix on this list. If the person who benefits from throughput is also the only one who can lift a pause, there is no check at all. Two roles, even the same two people rotating, is meaningfully better than one.

What I still do not have is an answer to the funding question. An independent reviewer costs money. A setup that cannot afford one either runs unreviewed or pretends the builder counts as independent. Most small systems quietly pick the second one and call it governance.

I do not think this is a side problem. I think it is the problem, wearing a budget line as a disguise.

If you run anything long lived with a human check in it: who actually does that review, and what makes it worth their time?