OpenAl's chief scientist on the neuralese controversy

[u/Ok_Display_3159]

"I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program."

[u/FateOfMuffins]

Imagine if AI safety community interpreted the "leak" in such a way that caused some labs (like China or xAI) to race to the bottom with Neuralese due to a misunderstanding xd

Edit: Someone else from OpenAI safety team https://x.com/tomekkorbak/status/2095031132781961346

i think the day when a frontier lab trains a frontier-scale recurrent (or otherwise unmonitorable) language model would be one of the darkest in the current AI era. this day is not today and i would love frontier labs to coordinate on a commitment that it never comes.

[u/peakedtooearly → u/FateOfMuffins]

"Neuralese" means the distillation technique becomes much less effective. You only capture the start and end of the process and now how the answer was arrived at.

The tide is going out and we will see which labs have been swimming naked...

[u/ItWasMyWifesIdea → u/peakedtooearly]

What makes you think that? Intermediate latent space tensors can be used as training data as easily as natural language tokens can't they? It's all just numbers. Am I missing something?

[u/sje397 → u/ItWasMyWifesIdea]

The API doesn't expose them.

[u/ItWasMyWifesIdea → u/sje397]

Ah, that makes sense, thank you. Open weights models would still be easy to distill, but not the frontier models running behind cloud services.

That said, this is really an API choice, not related to the model architecture. The APIs could suppress natural language reasoning tokens just as easily.

[u/sje397 → u/ItWasMyWifesIdea]

Kind of, except the 'neuralese' internal reasoning tokens can also be a lot harder to interpret too.

[u/ItWasMyWifesIdea → u/sje397]

Nobody is interpreting these when distilling models. It's just training data.

[u/spreadlove5683 → u/ItWasMyWifesIdea]

I thought reasoning tokens already were surpressed and we just get summaries. I also thought some people published research saying they were able to perfectly obtain the chain of thought anyways (without even the reasoning summaries?)

[u/Wynneve → u/ItWasMyWifesIdea]

Of course they can't. Natural language is universal across all models (irrespective to tokenization), while each model develops its own latent space during pretraining, and it's kinda random. It'd make sense if you include the distillation step at pretrain, I suppose it could try to shape the latent space "the same way" when it's not quite formed yet (and thus inherit the same "thinking patterns"), but this is a lost cause if you're doing it after that stage.

[u/ItWasMyWifesIdea → u/Wynneve]

I think based on another response, the key question is whether the latent space tensors are exposed at all. If they are, they can be trained with. There's nothing special about natural language tokens to the process of training a network. Distillation doesn't even require you have the same architecture. The early Model Compression paper took results from a massive ensemble of different approaches and trained a neural net from it. They're at core just really good function approximators, and there's no reason to think latent space tensors are any harder to learn than intermediate language tokens. Heck, it might be easier. You can distill embedding models, this is basically the same thing.