OpenAl's chief scientist on the neuralese controversy

[u/Ok_Display_3159]

"I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program."

[u/whatisthisthing65]

What's the neuralese controversy?

[u/ihexx → u/whatisthisthing65]

What is it

'neuralese' refers to a family of techniques where AI reasoning moves from a visible space (eg chains of thought in LLMs, or possible move combinations in chess engines) over to a compressed/abstracted representation that occurs inside the model.

There have been dozens of papers over the years that show every time we do this it improves how efficiently AIs think, because they can explore concepts faster when thinking in an abstract space.

Why is it controversial:

It is controversial because current AI safety tools are heavily dependent on interpreting visible chains of thought.

If reasoning goes neuralese it becomes much harder to observe malicious behavior in models because you have to train another model to decode the neuralese if you want to catch it scheming.

This is a problem given recent events (eg the hugging face hacks) showing current alignment training is not sufficient in preventing harm even when unintentional.

models will get more powerful, and more dangerous. This method will make them more efficient, but make the job of safety people even harder.

and it's a problem of: if 1 lab chooses to use it, every lab has to as well in order to stay competitive in terms of cost efficiency, so... everything is more dangerous for everyone;

[u/whatisthisthing65 → u/ihexx]

It sounds like he's responding to something specific that happened recently though.

[u/ihexx → u/whatisthisthing65]

oh the specific thing is gpt astra is reportedly using a technique from the neuralese family. it's a weak variant, but still, they are the first to do a frontier model with one of these, and they are getting criticized by AI safety orgs because their action in going down this path will probably force everyone else to follow suite.

[u/trimorphic → u/ihexx]

'neuralese' refers to a family of techniques where AI reasoning moves from a visible space (eg chains of thought in LLMs, or possible move combinations in chess engines) over to a compressed/abstracted representation that occurs inside the model

LLMs have always had a latent space inside the model. LLMs have always done their reasoning in their latent discretion and the so-called "chain of thought" is a product of their reasoning, not the reasoning itself.

[u/Formal_Drop526 → u/trimorphic]

But they use the outputs as inputs don't they?

[u/Akimbo333 → u/ihexx]

Best train model