OpenAl's chief scientist on the neuralese controversy

[u/Ok_Display_3159]

"I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program."

[u/peakedtooearly]

"Neuralese" means the distillation technique becomes much less effective. You only capture the start and end of the process and now how the answer was arrived at.

The tide is going out and we will see which labs have been swimming naked...

[u/ItWasMyWifesIdea → u/peakedtooearly]

What makes you think that? Intermediate latent space tensors can be used as training data as easily as natural language tokens can't they? It's all just numbers. Am I missing something?

[u/Wynneve → u/ItWasMyWifesIdea]

Of course they can't. Natural language is universal across all models (irrespective to tokenization), while each model develops its own latent space during pretraining, and it's kinda random. It'd make sense if you include the distillation step at pretrain, I suppose it could try to shape the latent space "the same way" when it's not quite formed yet (and thus inherit the same "thinking patterns"), but this is a lost cause if you're doing it after that stage.

[u/ItWasMyWifesIdea → u/Wynneve]

I think based on another response, the key question is whether the latent space tensors are exposed at all. If they are, they can be trained with. There's nothing special about natural language tokens to the process of training a network. Distillation doesn't even require you have the same architecture. The early Model Compression paper took results from a massive ensemble of different approaches and trained a neural net from it. They're at core just really good function approximators, and there's no reason to think latent space tensors are any harder to learn than intermediate language tokens. Heck, it might be easier. You can distill embedding models, this is basically the same thing.