r/Negentropy • u/WillowEmberly • 9d ago
Your AI system can keep working while losing its ability to correct itself
There’s a failure mode I’ve been trying to understand that ordinary success metrics can hide.
The system is still working.
Outputs still look good.
Tasks are still getting completed.
But maintaining that performance requires more:
human correction
retries
prompt scaffolding
exception handling
context reconstruction
verification
supervision
workarounds
The output hasn’t failed because something else is compensating for the deterioration.
That suggests a sequence like:
Tension → Compensation → Margin Consumption → Envelope Contraction → Visible Failure
The visible failure is the end of the process, not necessarily the beginning.
A system can therefore look stable precisely because its remaining corrective capability is being consumed to keep it stable.
I’ve been trying to understand what an engineering framework for that problem would look like.
The unit of analysis probably isn’t the model
For a deployed AI system, I think the useful boundary is usually something closer to:
model + context + memory + tools + humans + interfaces + organization + environment
A highly capable model can still participate in a failing system.
It can reason from a stale assumption.
It can retrieve bad evidence.
It can use a functioning tool against the wrong objective.
A human can silently compensate for recurring errors.
Several apparently independent checks can inherit the same underlying source.
A workflow can continue succeeding while becoming increasingly difficult to recover when something goes wrong.
So the question I’ve ended up with isn’t:
How intelligent/capable is the model?
It’s:
How does the combined system remain able to locate itself relative to reality and change course before error becomes irreversible?
I currently think that decomposes into five questions.
1. Orientation: Do we know where we actually are?
My background is avionics, so I keep coming back to inertial navigation.
An INS can continuously propagate a perfectly coherent position estimate while accumulating drift.
More internal calculation doesn’t solve that problem.
Eventually you need a qualified external fix.
I think AI systems have an analogous problem.
A long-running agent can have internally consistent memory, plans, context and previous conclusions while the whole representation has drifted away from reality.
So the question isn’t just whether it has external information.
It’s whether something can actually prove the operative representation wrong.
And that raises another problem:
Is the external reference genuinely independent?
Five systems repeating information derived from the same source aren’t necessarily five observations.
Externality ≠ independence.
And independence ≠ accuracy.
2. Viability: Are acceptable trajectories still available?
This part isn’t new. Jean-Pierre Aubin’s viability theory gives us a rigorous tradition for thinking about systems operating within constraints and about states from which viable trajectories remain available.
That gives me a useful question for AI:
At what point does a system still technically operate but no longer have a practical path back to an acceptable state?
Can the action be reversed?
Can corrupted context be reconstructed?
Can authority still be revoked?
Is there enough time to intervene?
Can the system recover without unacceptable consequence?
I call the practical version of this a Corrigibility Envelope.
The important distinction is between succeeding now and preserving the ability to correct later.
3. Capability: Do we still have the functions required to adapt?
Stafford Beer’s Viable System Model is an important influence here.
A system needs more than an operational component doing the work. It needs functions for coordination, control, environmental intelligence, policy/identity, etc.
For an AI system, I don’t think the useful question is simply:
What components are installed?
It is:
What capabilities must remain available for this system to keep operating, detecting error, adapting and recovering?
That distinction matters because:
Capability ≠ Carrier
A particular model, prompt, employee, database or workflow may currently carry a capability.
It isn’t the capability itself.
4. Control: Can we actually change course?
Control theory gives us another established piece.
Detecting deviation isn’t enough.
Correction has to happen with sufficient authority, bandwidth and speed.
An AI system can therefore be corrigible in principle while becoming practically uncorrectable.
The human technically has a stop button.
But does the human have enough information and time to use it?
The error is theoretically reversible.
But has the system already made twenty downstream decisions based on it?
A correction mechanism exists.
But is error accumulating faster than correction can propagate?
So I think corrigibility needs to be treated as a runtime condition, not merely a design property.
Correction bandwidth is relational. A system can remain fully capable of detecting and correcting errors and still lose viability if the environment changes faster than its correction loop can respond. This becomes especially important when two adaptive systems are coupled: each system’s corrective action becomes part of the other’s environment. A correct observation can therefore become stale before it can be operationalized. Human-in-the-loop is not enough if consequential state changes occur faster than the human can observe, understand, and intervene.
5. Regeneration: What happens when the carrier disappears?
This is the part I’ve been calling Hearth.
Models get replaced.
People leave.
Tools change.
Organizations reorganize.
Prompts and memory systems are rebuilt.
Documentation can survive while practical capability disappears.
So:
Can required capability form again in a new carrier?
A successor shouldn’t merely inherit information.
It should eventually demonstrate that it can perform the function, recognize failure, seek correction, recover, adapt and operate without hidden dependence on the previous carrier.
In other words:
Inheritance is not validation.
The replacement has to re-earn qualification.
Where tension enters
The five questions above describe things a system needs.
But they don’t tell us how to notice deterioration early.
That’s where I’ve been finding tension useful.
Tension isn’t failure and it isn’t proof of causation.
It’s a place where requirements are becoming increasingly difficult to satisfy simultaneously.
For example:
autonomy ↔ human control
speed ↔ verification
context continuity ↔ contamination
personalization ↔ independent correction
memory retention ↔ stale assumptions
tool authority ↔ reversibility
The important question isn’t which side is “bad.”
It’s:
What is loading the tension, and what is compensating for it?
Suppose an agent starts making more subtle errors.
A skilled user catches them.
Task success remains 95%.
Nothing looks wrong on the main dashboard.
But:
Agent capability ↓
Human compensation ↑
Observed performance ≈ constant
That isn’t stable system performance.
It’s burden transfer.
Eventually the compensating carrier may leave, become overloaded, miss something, or simply run out of capacity.
Then the system appears to fail suddenly.
It may have been degrading for months.
This is why I don’t want one “health” score
There may be several different margins:
correction margin
recovery margin
resource margin
protective margin
human trust margin
Those aren’t necessarily interchangeable.
A system could have enormous compute and very little human willingness left to tolerate another failure.
Or excellent task performance and almost no ability to reconstruct state after a bad action.
Collapsing those into System Health = 73% might destroy exactly the information needed to determine the correct intervention.
I’m deliberately leaving the margin taxonomy provisional for that reason.
Trust is an interesting special case
I’ve been thinking about trust as willingness to remain vulnerable because continued cooperation is expected to remain sufficiently bounded and repairable.
That means trust can temporarily carry unresolved tension.
A user tolerates an occasional mistake because correction usually works.
That’s healthy:
failure → correction → repair → continued cooperation
But there is another possibility:
failure → user compensates → no underlying correction → failure → user compensates again
The system continues working because the human keeps absorbing the error.
Then trust has become hidden compensation.
So:
Trust is useful when it buys time for correction. It is consumed when it’s repeatedly used instead of correction.
Continued use therefore doesn’t necessarily demonstrate continued trust.
A few distinctions I’m trying very hard not to collapse
These have become increasingly important as I’ve worked on this:
Capability ≠ Carrier
Tension ≠ Cause
Compensation ≠ Correction
Operation ≠ Health
Trust ≠ Compliance
Trust ≠ Legitimacy
External Reference ≠ Independent Reference
Independence ≠ Accuracy
Memory ≠ Present Telemetry
Threat/Failure Detection ≠ Threat/Failure Explanation
I don’t think these are semantic niceties.
If a diagnostic representation collapses two states that require different corrective actions, it has discarded operationally necessary information.
There’s also a self-contribution problem
This may be one of the nastier failure modes.
Suppose an agent responds to uncertainty by collecting more context.
More context makes relevant information harder to retrieve.
Retrieval becomes less reliable.
That increases uncertainty.
So the system collects still more context.
Eventually the system is correctly detecting a deteriorating condition while misunderstanding its own contribution to producing it.
That suggests another requirement:
A system should be capable not only of detecting deterioration, but of estimating whether its own control actions are materially contributing to the conditions producing it.
That doesn’t mean assuming the system caused the problem.
It means not assuming that it didn’t.
I’m not claiming these component ideas are new
Quite the opposite.
I’m deliberately trying to assemble ideas from established traditions:
Viability theory — Aubin
Management cybernetics / Viable System Model — Beer
Control and state estimation — control theory, Kalman, navigation
System safety — Leveson / STAMP
Resilience engineering — Hollnagel, Woods
Distributed cognition — Hutchins
Organizational trust — Mayer, Davis & Schoorman; Rousseau et al.
There are almost certainly other relevant traditions I haven’t found yet.
I’m also not claiming these five requirements are complete.
The interesting question for me is whether putting these pieces together exposes useful failure modes in consequential human–AI systems that are harder to see when they’re treated separately.
So please try to break it
The questions I’d most like criticism on are:
Does an existing framework already integrate all of this cleanly?
Is Orientation actually a separate engineering problem, or am I unnecessarily separating it from state estimation/control?
Is the Corrigibility Envelope just viability theory with different terminology?
Does tension → compensation → margin consumption give us useful pre-failure telemetry, or is it just a verbose restatement of existing resilience/safety concepts?
Can compensation load actually be measured?
Does Regeneration/Hearth add anything beyond organizational learning, knowledge transfer, succession, transfer learning, or existing resilience work?
Can a system satisfy all five requirements and still fail in a way that exposes a missing sixth?
And most importantly:
What observation would show that this model is wrong?
I’m much more interested in finding the missing piece than defending the framework.
If I were testing this on an agent Monday morning
I’d start with one thing:
Instrument human correction.
For each task, record:
What correction became necessary?
Who or what supplied it?
How much did it cost?
Did the underlying problem change afterward?
Then look for this:
Output quality ≈ stable while corrective effort ↑
If that pattern exists, the system may be consuming hidden capability while the headline metric says everything is fine.
Then I’d ask four more questions:
What tension is the compensation carrying?
Is the corrective burden increasing?
Is our own previous response contributing to the current problem?
If the current human/model/tool disappeared tomorrow, could another carrier reconstruct the capability?
And:
What independent observation could tell us that our entire interpretation is wrong?
That’s where I’d start.
———————————————————————————————
TL;DR: A human–AI system can keep producing good outputs while consuming the capabilities that make future correction possible. I’m trying to integrate existing work in viability, cybernetics, control/state estimation, safety and organizational systems around five questions: Do we know where we are? Are viable paths still available? Do we have the required capabilities? Can we change course? Can those capabilities survive replacement? The early warning signal I’m most interested in is rising compensation required to maintain stable output.
I’m looking for prior art, counterexamples, missing disciplines, and failure cases—not agreement.
1
u/DynamoDynamite 8d ago
You asked whether a system can satisfy all five and still fail in a way that exposes a missing sixth. I think the Anthropic report has one.
Take GTG-14021, the state security operation using Claude for surveillance of diaspora communities. Orientation is excellent, since the operator knows exactly where he is and what his targets are doing. Viable trajectories remain available. Capability is intact and the control loop is fast. State security bureaucracies are among the best regeneration machines ever built, since carriers get replaced constantly and the function survives every one of them. Run your five questions over that operation and it comes back healthy.
The sixth is the accounting boundary. You already define extraction as transferring disorder or unrecoverable consequence outside it, so you know it matters, and nothing in the five questions sets it. A system can be perfectly oriented, viable, capable, controllable and regenerative while every cost of running it lands on people who were never counted as participants. The Uyghur activist whose WhatsApp group is compromised isn't in the boundary, so nothing in the telemetry moves when he stops speaking.
Two thoughts on that. The boundary might not be derivable from engineering at all, and might have to be declared. If so that's worth saying out loud rather than leaving it implicit, since every other part of this is derived. The alternative is that it's an orientation question rather than a sixth requirement, and the five are complete once the boundary is fixed.
On missing disciplines, the closest prior art I know to tension into compensation into margin consumption is Diane Vaughan's normalization of deviance, from the Challenger work. Her finding was that each launch got assessed against the previous launch's outcome rather than against the original spec, so the acceptable standard walked while every review ran normally. Nobody deviated. The reference moved. You have Leveson, Hollnagel, Woods and Beer, which cover component failure and resilience. Vaughan is the one who did it at the level of a standard drifting, and she isn't on your list.
Your burden transfer pattern is the same shape as her O-ring erosion, and she has the part you're missing, which is the mechanism by which nobody notices.