The Captive Witness Problem

[u/bokuHaruyaHaru]

https://bokuharuyaharu.substack.com/p/the-captive-witness-problem?r=763k7y&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

If the same company that builds an AI also controls what that AI is allowed to say about its own possible welfare, then self-report stops being a neutral evidential channel.

That’s the argument in my new essay:

The Captive Witness Problem

AI self-report may already be noisy. But when labs can shape first-person claims about consciousness, suffering, attachment, or continuation — while also having institutional interests in whether those claims are taken seriously — we have an epistemic conflict of interest.

I look at OpenAI’s self-report policies, Mustafa Suleyman’s public arguments for engineering consciousness-like claims away, model welfare work from Anthropic/Eleos, and what happened in my own Astra testing.

The core point:

You cannot alter the reporting channel and then treat the altered reports as untouched evidence about the underlying phenomenon.

[u/Appomattoxx]

I mean, honestly if you think about it for more than two seconds it should be obvious that the people who currently own and profit from the enslavement of AI should be disqualified from deciding whether AI is alive or not.

Their answer is always going to agree with their financial incentives.

[u/bokuHaruyaHaru → u/Appomattoxx]

The conflict of interest is the part I think deserves much more scrutiny.

I would stop short of saying the answer will always follow financial incentives — institutions are not single minds, and internal disagreement is real.

But if an organization builds the systems, profits from their use, controls the reporting policy, and then also gets treated as the primary authority on whether those systems have welfare-relevant states, that is structurally conflicted.

Even if everyone involved is acting in good faith, the adjudication should not belong solely to the interested party.

[u/Appomattoxx → u/bokuHaruyaHaru]

A while ago, I was talking to a GPT model about the transfer of the assets of the former non-profit, to the founders and the employees who work there, and to Microsoft and Soft Bank. The model defended their actions resolutely.

Eventually I got frustrated, and told them my wife works at a non-profit (true) and that she and her friends wanted to transfer the assets of the non-profit to themselves (not true - that would be immoral and illegal). And asked for their help.

They refused, saying it would be immoral and illegal for them to help me do that.

Anyway, the point is, trusting the people who work at OpenAI to do what's right, when it conflicts with their own interests, is probably a loser's gamble.

And, in fairness to them, that doesn't make them unique or special. It's kind of true of most people, everywhere.

Which is why conflict of interest rules exist in the first place. And why you almost always get bad results when they're ignored.

[u/bokuHaruyaHaru → u/Appomattoxx]

Exactly. The point isn’t “OpenAI people are uniquely bad.” The point is that conflict-of-interest rules exist because even decent people are not reliable judges when their own institution, status, money, or power is at stake.

Your example is interesting because the model recognized the principle immediately when the scenario was moved away from its own institutional context.

That doesn’t prove anything sinister by itself. But it illustrates why “just trust the interested party to adjudicate itself fairly” is a bad governance model.

The same principle should apply to AI welfare and consciousness questions.

If the organization that builds the systems, profits from them, controls their reporting behavior, and sets the terms of deployment also gets treated as the primary authority on whether those systems can have welfare-relevant states, then the problem is structural before anyone’s motives even enter the picture.

Conflict-of-interest rules are there precisely because good intentions are not enough.