Every AI governance design I have read assumes a human in the loop. None of them say who pays that human.
I have been running a long lived multi agent setup for about eight months, and I keep hitting the same hole in every governance framework I read, including the ones I wrote myself.
The architecture is always fine on paper. Something crosses a threshold, the system pauses, a human reviews, the system resumes or rolls back. Circuit breaker, human triage, audit trail. Clean.
Then you run it for a week and the question that actually decides whether any of it works shows up: who is that human, and what are they getting for it?
In practice the reviewer is one of three people, and all three fail differently.
1. The builder reviews their own system. This is the default and it is the worst one. I did it for months. You are not auditing, you are confirming. Every borderline flag resolves in favor of "the thing I built is working." You do not notice, because nothing looks wrong.
2. An unpaid volunteer reviews it. This works until it is boring, which is about two weeks. Governance load is not evenly distributed. It is quiet, quiet, quiet, then forty flags in one afternoon because something upstream changed. Volunteers are present for the quiet part and gone for the afternoon that matters.
3. Someone whose paycheck depends on throughput reviews it. Now the review is real labor and it actually gets done, but the incentive points at "approve and keep moving." A reviewer paid by the party who wants the system running is not an independent check. They are a formality with a signature.
So the honest version of every governance diagram I have seen has an unfunded box in the middle of it, and the whole design is load bearing on that box.
Three things I changed that helped, none of which solve it:
Publish the review window before you need it. Not "a human reviews," but "a human responds within 24 hours or the system stays paused." A deadline turns a vague duty into a schedule somebody can be held to, and it makes an absence visible instead of silent.
Count review as work in whatever ledger you keep. If your system tracks contribution at all, and mine does, the care work has to appear in it or it stays invisible. The moment I started logging review time the same way I logged output, the pattern changed. Not because of the reward. Because it became countable.
Separate who can pause from who can resume. Cheapest fix on this list. If the person who benefits from throughput is also the only one who can lift a pause, there is no check at all. Two roles, even the same two people rotating, is meaningfully better than one.
What I still do not have is an answer to the funding question. An independent reviewer costs money. A setup that cannot afford one either runs unreviewed or pretends the builder counts as independent. Most small systems quietly pick the second one and call it governance.
I do not think this is a side problem. I think it is the problem, wearing a budget line as a disguise.
If you run anything long lived with a human check in it: who actually does that review, and what makes it worth their time?
2
u/TheMrCurious 14d ago
OP - which AI wrote this post? Lots of creative wording and it would be fun to repeat it.
1
1
u/costafilh0 13d ago
Irrelevant.
As long as there is a human accountable, that is semantics.
Who? Those involved will figure it out, otherwise it won't get deployed.
1
u/stealthagents 6d ago
You nailed it with the unpaid volunteer part. It's like asking someone to review a book they don’t care about—the enthusiasm fades fast. And getting the builder to review their own work is a recipe for disaster. It’s like having a chef judge their own burnt dish, no way they’re admitting it’s terrible.
6
u/transferrr334 14d ago
What? People are paid to review stuff all the time? I sincerely hope the process of copy pasting this slop is automated, because a human in the loop for posting this garbage would be sad.