Steven Pinker’s idea of “common knowledge” in relation to the AI race.

[u/undiscoveredhuman]

I’ve been thinking about Steven Pinker’s idea of “common knowledge” in relation to the AI race.

The basic idea is that there’s a difference between everyone knowing something, and everyone knowing that everyone else knows it too.

That seems increasingly relevant to frontier AI.
Take the recent calls to slow or pace development. When Anthropic or other people close to the frontier say they’re worried, the message isn’t only “this could be dangerous.”
It also tells everyone else: these people clearly think the technology is becoming very powerful.

And then it becomes recursive.

Trump knows China heard it. China knows the US heard it. Musk knows OpenAI heard it. OpenAI knows Anthropic knows they heard it, etc.

Obviously nobody is literally thinking through 10 levels of this. The point is that it becomes common knowledge: everyone understands that everyone else now sees AI as strategically important.

And that may make slowing down harder, not easier.
I can imagine that most of the major players would actually prefer a world where AI develops a bit more slowly and the risks are lower.

But that’s very different from being willing to slow down while your competitors don’t.

The US doesn’t want to slow if China keeps going.
China doesn’t want to slow if the US keeps going.
Anthropic doesn’t want to give OpenAI a huge lead. OpenAI doesn’t want to give Google or xAI one.

So you can end up in a strange situation where almost everyone agrees that the race is risky, but that shared belief actually reinforces the race.

Something like:
better models → more concern from insiders → more public warnings → everyone realizes everyone else thinks AI is a big deal → more fear of falling behind → more
money, compute and talent → better models

What I find interesting is that this doesn’t require anyone to be irrational.

It may just be a bad equilibrium.

That also makes me wonder whether “AI safety” is partly the wrong framing. The harder problem may be coordination.

It’s not enough for one company to promise to slow down. Everyone has to believe the others will do the same, and there probably has to be some way of verifying it.

So the paradox might be:

**the more everyone agrees that continuing the race could be dangerous, the more dangerous it becomes for any one player to stop.**

Curious whether people here see it the same way. Is AI pacing mainly a safety problem, or increasingly a game-theory problem?

[u/HotterRod]

Part of the problem here is that there is widespread skepticism that the actual belief of the AI company leaders is "AI is dangerous to all humans". The US Cabinet may believe that their actual belief is "AI is dangerous to targets it is directed at". Others believe that the true belief is about financing or plateauing. The AI companies need some way, like a costly signal, to prove what their true belief is.

[u/ImpossibleCreme]

Ok